[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:26:56 dist-test-slave-jm5g krb5kdc[17365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:26:56 dist-test-slave-jm5g krb5kdc[17365](info): set up 2 sockets
Jun 18 22:26:56 dist-test-slave-jm5g krb5kdc[17365](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260618 22:26:57.703191 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:26:57 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:26:57.817459 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33729
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:26:57.924232 17378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:26:57.924504 17378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:26:57.924561 17378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:26:57.928524 17378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:26:57.928601 17378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:26:57.928627 17378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:26:57.928648 17378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:26:57.928668 17378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:26:57.934000 17378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:33729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:26:57.935134 17378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:26:57.936204 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:26:57.944963 17384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:57.944978 17378 server_base.cc:1034] running on GCE node
W20260618 22:26:57.944897 17383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:26:57.944897 17386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:57.945683 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:26:57.946892 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:26:57.948081 17378 hybrid_clock.cc:648] HybridClock initialized: now 1781821617948056 us; error 34 us; skew 500 ppm
Jun 18 22:26:57 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821617, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:26:57.950771 17378 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:26:57.951715 17378 webserver.cc:460] Webserver started at http://127.16.243.190:44273/ using document root <none> and password file <none>
I20260618 22:26:57.952255 17378 fs_manager.cc:359] Metadata directory not provided
I20260618 22:26:57.952328 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:26:57.952521 17378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:26:57.954668 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "ef62db68f52b4457add3e1ef47c66829"
format_stamp: "Formatted at 2026-06-18 22:26:57 on dist-test-slave-jm5g"
server_key: "fec70e54a1b87d08cebdf8489832da25"
server_key_iv: "dbb4e7209b4d8a4960a928e7dd2e0153"
server_key_version: "encryptionkey@0"
I20260618 22:26:57.955214 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "ef62db68f52b4457add3e1ef47c66829"
format_stamp: "Formatted at 2026-06-18 22:26:57 on dist-test-slave-jm5g"
server_key: "fec70e54a1b87d08cebdf8489832da25"
server_key_iv: "dbb4e7209b4d8a4960a928e7dd2e0153"
server_key_version: "encryptionkey@0"
I20260618 22:26:57.958989 17378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:26:57.961493 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:57.962838 17378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:26:57.962955 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "ef62db68f52b4457add3e1ef47c66829"
format_stamp: "Formatted at 2026-06-18 22:26:57 on dist-test-slave-jm5g"
server_key: "fec70e54a1b87d08cebdf8489832da25"
server_key_iv: "dbb4e7209b4d8a4960a928e7dd2e0153"
server_key_version: "encryptionkey@0"
I20260618 22:26:57.963055 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:26:57.975178 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:26:57.978318 17378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:26:57.978472 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:26:57.985136 17378 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:33729
I20260618 22:26:57.985176 17445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:33729 every 8 connection(s)
I20260618 22:26:57.986102 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:26:57.988972 17446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:26:57.993875 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17378
I20260618 22:26:57.994001 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:26:57.994699 17446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829: Bootstrap starting.
I20260618 22:26:57.995335 17358 external_mini_cluster.cc:1428] Setting key d4ed247e8b925722e497d262b218f00f
I20260618 22:26:57.997258 17446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829: Neither blocks nor log segments found. Creating new log.
I20260618 22:26:57.998068 17446 log.cc:824] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829: Log is configured to *not* fsync() on all Append() calls
I20260618 22:26:58.000054 17446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829: No bootstrap required, opened a new log
I20260618 22:26:58.003901 17446 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } }
I20260618 22:26:58.004138 17446 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:26:58.004194 17446 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef62db68f52b4457add3e1ef47c66829, State: Initialized, Role: FOLLOWER
I20260618 22:26:58.004647 17446 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } }
I20260618 22:26:58.004779 17446 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:26:58.004845 17446 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:26:58.004942 17446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:26:58.005844 17446 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } }
I20260618 22:26:58.006166 17446 leader_election.cc:302] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef62db68f52b4457add3e1ef47c66829; no voters:
I20260618 22:26:58.006466 17446 leader_election.cc:288] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:26:58.006577 17451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:26:58.006745 17451 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [term 1 LEADER]: Becoming Leader. State: Replica: ef62db68f52b4457add3e1ef47c66829, State: Running, Role: LEADER
I20260618 22:26:58.007059 17451 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } }
I20260618 22:26:58.007795 17446 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:26:58.008744 17453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef62db68f52b4457add3e1ef47c66829. Latest consensus state: current_term: 1 leader_uuid: "ef62db68f52b4457add3e1ef47c66829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } } }
I20260618 22:26:58.008867 17453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [sys.catalog]: This master's current role is: LEADER
I20260618 22:26:58.009130 17452 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef62db68f52b4457add3e1ef47c66829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62db68f52b4457add3e1ef47c66829" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33729 } } }
I20260618 22:26:58.009202 17452 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829 [sys.catalog]: This master's current role is: LEADER
I20260618 22:26:58.009240 17460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:26:58.012594 17460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:26:58.018589 17460 catalog_manager.cc:1347] Generated new cluster ID: b87540d3cf684105876ecb0577a490e0
I20260618 22:26:58.018654 17460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:26:58.020951 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:57.998203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.16.243.190:33729)
0618 22:26:57.998665 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:57.998684 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:57.999909 (+ 1225us) server_negotiation.cc:407] Connection header received
0618 22:26:58.003511 (+ 3602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.003531 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.003901 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.004236 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.006736 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.008032 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.009784 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.010494 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.011585 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.015127 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.015133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.015146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.015173 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.016957 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.017606 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.017613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.017619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.017779 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.018188 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.018192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.018194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.018765 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.018990 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.019808 (+ 818us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.020082 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260618 22:26:58.035818 17460 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:26:58.037086 17460 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:26:58.056051 17460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef62db68f52b4457add3e1ef47c66829: Generated new TSK 0
I20260618 22:26:58.056792 17460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:26:58.089625 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:26:58.215325 17472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:26:58.215598 17472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:26:58.215642 17472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:26:58.219563 17472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:26:58.219624 17472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:26:58.219683 17472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:26:58.224934 17472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:26:58.225993 17472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:26:58.226997 17472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:26:58.236225 17480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:26:58.236233 17477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:26:58.236366 17478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:58.236661 17472 server_base.cc:1034] running on GCE node
I20260618 22:26:58.237038 17472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:26:58.237643 17472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:26:58.238791 17472 hybrid_clock.cc:648] HybridClock initialized: now 1781821618238762 us; error 43 us; skew 500 ppm
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:26:58.241430 17472 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:26:58.242324 17472 webserver.cc:460] Webserver started at http://127.16.243.129:36261/ using document root <none> and password file <none>
I20260618 22:26:58.242905 17472 fs_manager.cc:359] Metadata directory not provided
I20260618 22:26:58.242972 17472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:26:58.243160 17472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:26:58.245157 17472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "d73ff5c9910b4ac1b7f5967430de1970"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9873e60e0562d8d22ba8c9190f7988b9"
server_key_iv: "79f1820f155b9599d29680c447240d63"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.245679 17472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "d73ff5c9910b4ac1b7f5967430de1970"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9873e60e0562d8d22ba8c9190f7988b9"
server_key_iv: "79f1820f155b9599d29680c447240d63"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.249403 17472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:26:58.251809 17487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.253013 17472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:26:58.253118 17472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "d73ff5c9910b4ac1b7f5967430de1970"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9873e60e0562d8d22ba8c9190f7988b9"
server_key_iv: "79f1820f155b9599d29680c447240d63"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.253238 17472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:26:58.273535 17472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:26:58.276988 17472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:26:58.277170 17472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:26:58.277757 17472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:26:58.278618 17472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:26:58.278664 17472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.278716 17472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:26:58.278745 17472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.287222 17472 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:37241
I20260618 22:26:58.287259 17600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:37241 every 8 connection(s)
I20260618 22:26:58.288106 17472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:26:58.296824 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17472
I20260618 22:26:58.296927 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:26:58.297148 17358 external_mini_cluster.cc:1428] Setting key b259cc242f48f2f80182e3332553a293
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:26:58.301538 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.289834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:48525 (local address 127.16.243.190:33729)
0618 22:26:58.289961 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:58.289965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:58.290822 (+ 857us) server_negotiation.cc:407] Connection header received
0618 22:26:58.292102 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.292106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.292167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.292244 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.293443 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.293972 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.295196 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.295423 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.295513 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.298029 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.298032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.298035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.298059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.299495 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.300001 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.300005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.300007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.300087 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.300405 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.300408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.300410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.300597 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.300680 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.301296 (+ 616us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.301414 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:26:58.302703 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.290205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33729 (local address 127.16.243.129:48525)
0618 22:26:58.290663 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0618 22:26:58.290705 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:26:58.291888 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:26:58.292441 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:26:58.292448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:26:58.292809 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:26:58.293297 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.293311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.294094 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.294096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.295068 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.295078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.295569 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.295573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.295737 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.296323 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:26:58.296341 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:26:58.297861 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:26:58.299611 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.299617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.299636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.299906 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.300169 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.300172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.300174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.300315 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.300697 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:26:58.300704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:26:58.301005 (+ 301us) client_negotiation.cc:769] Sending connection context
0618 22:26:58.301214 (+ 209us) client_negotiation.cc:240] Negotiation successful
0618 22:26:58.301572 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260618 22:26:58.304023 17601 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33729
I20260618 22:26:58.304298 17601 heartbeater.cc:464] Registering TS with master...
I20260618 22:26:58.304922 17601 heartbeater.cc:511] Master 127.16.243.190:33729 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:26:58.306586 17410 ts_manager.cc:194] Registered new tserver with Master: d73ff5c9910b4ac1b7f5967430de1970 (127.16.243.129:37241)
I20260618 22:26:58.307796 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:48525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:26:58.321342 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:26:58.432484 17606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:26:58.432746 17606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:26:58.432792 17606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:26:58.436913 17606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:26:58.436976 17606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:26:58.437038 17606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:26:58.444059 17606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:26:58.445777 17606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:26:58.447415 17606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:26:58.460510 17611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:26:58.460510 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:58.460736 17606 server_base.cc:1034] running on GCE node
W20260618 22:26:58.460803 17614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:58.461524 17606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:26:58.462386 17606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:26:58.463570 17606 hybrid_clock.cc:648] HybridClock initialized: now 1781821618463531 us; error 52 us; skew 500 ppm
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:26:58.467044 17606 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:26:58.468094 17606 webserver.cc:460] Webserver started at http://127.16.243.130:37393/ using document root <none> and password file <none>
I20260618 22:26:58.468789 17606 fs_manager.cc:359] Metadata directory not provided
I20260618 22:26:58.468868 17606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:26:58.469098 17606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:26:58.472023 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "494fb7ca84ab4aa4815a6f713992125c"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "6b9d81ff5a79df122bfe5c2e46f4681f"
server_key_iv: "e935a0162cab5f63df85c305579cef83"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.472770 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "494fb7ca84ab4aa4815a6f713992125c"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "6b9d81ff5a79df122bfe5c2e46f4681f"
server_key_iv: "e935a0162cab5f63df85c305579cef83"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.478114 17606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260618 22:26:58.481858 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.483668 17606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:26:58.483807 17606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "494fb7ca84ab4aa4815a6f713992125c"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "6b9d81ff5a79df122bfe5c2e46f4681f"
server_key_iv: "e935a0162cab5f63df85c305579cef83"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.483953 17606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:26:58.504822 17606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:26:58.508240 17606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:26:58.508440 17606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:26:58.509089 17606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:26:58.509967 17606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:26:58.510015 17606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.510061 17606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:26:58.510087 17606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.519058 17606 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:33113
I20260618 22:26:58.519078 17734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:33113 every 8 connection(s)
I20260618 22:26:58.520028 17606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:26:58.528046 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17606
I20260618 22:26:58.528139 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:26:58.528358 17358 external_mini_cluster.cc:1428] Setting key 41b7abd57053f53801d476046cde4235
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:26:58.535709 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.522037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39543 (local address 127.16.243.190:33729)
0618 22:26:58.522156 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:58.522160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:58.523081 (+ 921us) server_negotiation.cc:407] Connection header received
0618 22:26:58.524437 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.524441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.524496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.524587 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.526252 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.527114 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.528577 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.528891 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.528994 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.532164 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.532168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.532171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.532198 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.533596 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.534099 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.534102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.534104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.534184 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.534508 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.534512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.534514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.534750 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.534838 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.535397 (+ 559us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.535538 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:26:58.536715 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.522422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33729 (local address 127.16.243.130:39543)
0618 22:26:58.522929 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0618 22:26:58.522973 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:26:58.524162 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:26:58.524818 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:26:58.524829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:26:58.525363 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:26:58.526076 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.526097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.527264 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.527267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.528444 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.528454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.529176 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.529180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.529405 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.530085 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:26:58.530106 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:26:58.531978 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:26:58.533714 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.533721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.533740 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.533998 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.534272 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.534275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.534277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.534419 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.534852 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:26:58.534860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:26:58.535118 (+ 258us) client_negotiation.cc:769] Sending connection context
0618 22:26:58.535313 (+ 195us) client_negotiation.cc:240] Negotiation successful
0618 22:26:58.535608 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":188,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:26:58.537928 17735 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33729
I20260618 22:26:58.538192 17735 heartbeater.cc:464] Registering TS with master...
I20260618 22:26:58.538739 17735 heartbeater.cc:511] Master 127.16.243.190:33729 requested a full tablet report, sending...
I20260618 22:26:58.539788 17410 ts_manager.cc:194] Registered new tserver with Master: 494fb7ca84ab4aa4815a6f713992125c (127.16.243.130:33113)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:26:58.540414 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:39543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:26:58.553752 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:26:58.664951 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:26:58.665231 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:26:58.665275 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:26:58.669462 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:26:58.669528 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:26:58.669588 17740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:26:58.675060 17740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:26:58.676117 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:26:58.677102 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:26:58.686578 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:26:58.686602 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:58.686602 17740 server_base.cc:1034] running on GCE node
W20260618 22:26:58.686580 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:26:58.687279 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:26:58.687989 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:26:58.689157 17740 hybrid_clock.cc:648] HybridClock initialized: now 1781821618689130 us; error 41 us; skew 500 ppm
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:26:58.692044 17740 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:26:58.692934 17740 webserver.cc:460] Webserver started at http://127.16.243.131:33697/ using document root <none> and password file <none>
I20260618 22:26:58.693481 17740 fs_manager.cc:359] Metadata directory not provided
I20260618 22:26:58.693548 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:26:58.693738 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:26:58.695907 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "3feee2aa8fee4b068dcca41456c50498"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9284eaf52fbc7b14939dbf96f3f1c356"
server_key_iv: "e8021ed9ad4469d7f4d035a695061628"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.696471 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "3feee2aa8fee4b068dcca41456c50498"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9284eaf52fbc7b14939dbf96f3f1c356"
server_key_iv: "e8021ed9ad4469d7f4d035a695061628"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.700340 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:26:58.702970 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.704298 17740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:26:58.704420 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "3feee2aa8fee4b068dcca41456c50498"
format_stamp: "Formatted at 2026-06-18 22:26:58 on dist-test-slave-jm5g"
server_key: "9284eaf52fbc7b14939dbf96f3f1c356"
server_key_iv: "e8021ed9ad4469d7f4d035a695061628"
server_key_version: "encryptionkey@0"
I20260618 22:26:58.704536 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:26:58.730374 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:26:58.733692 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:26:58.733888 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:26:58.734488 17740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:26:58.735356 17740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:26:58.735404 17740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.735455 17740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:26:58.735484 17740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:26:58.744071 17740 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:43431
I20260618 22:26:58.744096 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:43431 every 8 connection(s)
I20260618 22:26:58.744971 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:26:58.750146 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17740
I20260618 22:26:58.750257 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:26:58.750515 17358 external_mini_cluster.cc:1428] Setting key b8aec0df0596513eb9b795bcd9dbe97c
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:26:58.758883 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.746755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:60243 (local address 127.16.243.190:33729)
0618 22:26:58.746898 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:58.746903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:58.747786 (+ 883us) server_negotiation.cc:407] Connection header received
0618 22:26:58.749078 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.749081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.749127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.749199 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.750339 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.751005 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.752534 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.752841 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.752946 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.755359 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.755362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.755365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.755390 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.756662 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.757269 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.757272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.757273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.757363 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.757684 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.757687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.757688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.757966 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.758072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.758648 (+ 576us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.758756 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:26:58.759680 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.747182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33729 (local address 127.16.243.131:60243)
0618 22:26:58.747629 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0618 22:26:58.747671 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:26:58.748884 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:26:58.749353 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:26:58.749360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:26:58.749709 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:26:58.750182 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.750195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.751157 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.751161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.752425 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:26:58.752434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.752974 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.752977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:26:58.753168 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.753707 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:26:58.753723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:26:58.755174 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:26:58.756859 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.756865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.756884 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.757161 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.757440 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:26:58.757442 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:26:58.757444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:26:58.757576 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:26:58.758122 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:26:58.758127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:26:58.758384 (+ 257us) client_negotiation.cc:769] Sending connection context
0618 22:26:58.758568 (+ 184us) client_negotiation.cc:240] Negotiation successful
0618 22:26:58.758858 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260618 22:26:58.760929 17869 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33729
I20260618 22:26:58.761183 17869 heartbeater.cc:464] Registering TS with master...
I20260618 22:26:58.761682 17869 heartbeater.cc:511] Master 127.16.243.190:33729 requested a full tablet report, sending...
I20260618 22:26:58.762765 17410 ts_manager.cc:194] Registered new tserver with Master: 3feee2aa8fee4b068dcca41456c50498 (127.16.243.131:43431)
I20260618 22:26:58.763350 17410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:60243
I20260618 22:26:58.764897 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:26:58.789677 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.780507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.16.243.129:37241)
0618 22:26:58.780824 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:58.780837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:58.780863 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:26:58.780915 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.780920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.781103 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.781221 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.781839 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.782654 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.783448 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.783807 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.783882 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.786169 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.786180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.786193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.786220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.787865 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.788332 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.788343 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.788344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.788436 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.788692 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.788695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.788696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.788966 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.789201 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.789385 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.789518 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
W20260618 22:26:58.790663 17515 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33268
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:26:58.815315 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:26:58.807234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.16.243.129:37241)
0618 22:26:58.807387 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:26:58.807392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:26:58.807524 (+ 132us) server_negotiation.cc:407] Connection header received
0618 22:26:58.807676 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:26:58.807679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:26:58.807724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:26:58.807794 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:26:58.808497 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.809064 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.809971 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:26:58.810197 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:26:58.810271 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:26:58.812640 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:26:58.812643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:26:58.812645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:26:58.812671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:26:58.813848 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.814220 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.814223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.814225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.814302 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:26:58.814496 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:26:58.814498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:26:58.814500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:26:58.814703 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:26:58.814828 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:26:58.815056 (+ 228us) server_negotiation.cc:299] Negotiation successful
0618 22:26:58.815157 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:26:58.816358 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17472
I20260618 22:26:58.820932 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17606
I20260618 22:26:58.825208 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17740
I20260618 22:26:58.829694 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17378
2026-06-18T22:26:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2102 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17883](info): set up 2 sockets
Jun 18 22:26:58 dist-test-slave-jm5g krb5kdc[17883](info): commencing operation
krb5kdc: starting...
W20260618 22:26:59.782032 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:26:59 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:26:59.896987 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:38897
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:00.006261 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:00.006616 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:00.006685 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:00.010766 17896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:00.010844 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:00.010874 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:00.010895 17896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:00.010915 17896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:00.016402 17896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:38897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:00.017508 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:00.018543 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:00.026770 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.026819 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.026786 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:00.027215 17896 server_base.cc:1034] running on GCE node
I20260618 22:27:00.027765 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:00.028896 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:00.030056 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781821620030037 us; error 31 us; skew 500 ppm
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:00.032778 17896 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:00.033715 17896 webserver.cc:460] Webserver started at http://127.16.243.190:45479/ using document root <none> and password file <none>
I20260618 22:27:00.034240 17896 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:00.034297 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:00.034480 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:00.036586 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "c5a40e0f35fc42ca87b4303537b65827"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "6652444fc49bdd213db082b680f3463d"
server_key_iv: "b521e7c57a87928fdc312d4d903400d1"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.037113 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "c5a40e0f35fc42ca87b4303537b65827"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "6652444fc49bdd213db082b680f3463d"
server_key_iv: "b521e7c57a87928fdc312d4d903400d1"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.040997 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260618 22:27:00.043558 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.044893 17896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:00.044999 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "c5a40e0f35fc42ca87b4303537b65827"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "6652444fc49bdd213db082b680f3463d"
server_key_iv: "b521e7c57a87928fdc312d4d903400d1"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.045094 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:00.059913 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:00.063148 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:00.063306 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:00.070050 17896 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:38897
I20260618 22:27:00.070067 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:38897 every 8 connection(s)
I20260618 22:27:00.071050 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:00.073261 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17896
I20260618 22:27:00.073369 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:00.073635 17358 external_mini_cluster.cc:1428] Setting key 4c786e65eeb1f70b179aa89caad96c17
I20260618 22:27:00.074072 17964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:00.081799 17964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827: Bootstrap starting.
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:00.084239 17964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:00.085158 17964 log.cc:824] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:00.087288 17964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827: No bootstrap required, opened a new log
I20260618 22:27:00.089107 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.075273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.16.243.190:38897)
0618 22:27:00.075811 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.075832 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.075895 (+ 63us) server_negotiation.cc:407] Connection header received
0618 22:27:00.077001 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.077026 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.077490 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.077909 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:00.078538 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.080027 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.080891 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.081582 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.081746 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.084304 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:00.084311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:00.084329 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:00.084366 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:00.086271 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.086781 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.086786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.086791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.086920 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.087197 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.087200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.087201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.087633 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:00.087808 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.088100 (+ 292us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.088376 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":138,"threads_started":1}
I20260618 22:27:00.090909 17964 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } }
I20260618 22:27:00.091147 17964 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:00.091220 17964 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a40e0f35fc42ca87b4303537b65827, State: Initialized, Role: FOLLOWER
I20260618 22:27:00.091639 17964 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } }
I20260618 22:27:00.091773 17964 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:00.091838 17964 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:00.091938 17964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:00.092842 17964 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } }
I20260618 22:27:00.093174 17964 leader_election.cc:302] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5a40e0f35fc42ca87b4303537b65827; no voters:
I20260618 22:27:00.093477 17964 leader_election.cc:288] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:00.093578 17969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:00.093822 17969 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [term 1 LEADER]: Becoming Leader. State: Replica: c5a40e0f35fc42ca87b4303537b65827, State: Running, Role: LEADER
I20260618 22:27:00.094199 17969 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } }
I20260618 22:27:00.094771 17964 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:00.095845 17971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5a40e0f35fc42ca87b4303537b65827. Latest consensus state: current_term: 1 leader_uuid: "c5a40e0f35fc42ca87b4303537b65827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } } }
I20260618 22:27:00.095983 17971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:00.096201 17970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5a40e0f35fc42ca87b4303537b65827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a40e0f35fc42ca87b4303537b65827" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 38897 } } }
I20260618 22:27:00.096342 17970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:00.096395 17978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:00.099197 17978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:00.104781 17978 catalog_manager.cc:1347] Generated new cluster ID: a19d435aa1cf4e48aad177940deee32e
I20260618 22:27:00.104874 17978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:00.115142 17978 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:00.116154 17978 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:00.125303 17978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5a40e0f35fc42ca87b4303537b65827: Generated new TSK 0
I20260618 22:27:00.126053 17978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:00.156735 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:00.269779 17990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:00.270079 17990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:00.270125 17990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:00.274356 17990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:00.274437 17990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:00.274501 17990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:00.279989 17990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:00.281134 17990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:00.282209 17990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:00.291889 17996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.291889 17995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.291942 17998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:00.292090 17990 server_base.cc:1034] running on GCE node
I20260618 22:27:00.292665 17990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:00.293363 17990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:00.294585 17990 hybrid_clock.cc:648] HybridClock initialized: now 1781821620294537 us; error 35 us; skew 500 ppm
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:00.297603 17990 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:00.298668 17990 webserver.cc:460] Webserver started at http://127.16.243.129:41829/ using document root <none> and password file <none>
I20260618 22:27:00.299253 17990 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:00.299316 17990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:00.299517 17990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:00.301681 17990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "60ccd0a1ed6448bbaa4cd640d833891e"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "eb5fe1480f54614f237d696e2bb5421f"
server_key_iv: "c946cbbfd3f0cb548885d45812e732e0"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.302268 17990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "60ccd0a1ed6448bbaa4cd640d833891e"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "eb5fe1480f54614f237d696e2bb5421f"
server_key_iv: "c946cbbfd3f0cb548885d45812e732e0"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.306259 17990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260618 22:27:00.308890 18005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.310168 17990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:00.310284 17990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "60ccd0a1ed6448bbaa4cd640d833891e"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "eb5fe1480f54614f237d696e2bb5421f"
server_key_iv: "c946cbbfd3f0cb548885d45812e732e0"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.310412 17990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:00.341707 17990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:00.345079 17990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:00.345288 17990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:00.345907 17990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:00.346818 17990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:00.346865 17990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.346907 17990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:00.346936 17990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.355559 17990 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:45165
I20260618 22:27:00.355597 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:45165 every 8 connection(s)
I20260618 22:27:00.356487 17990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:00.363473 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 17990
I20260618 22:27:00.363575 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:00.363806 17358 external_mini_cluster.cc:1428] Setting key c175cb62257e4b6509574344019f6835
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:00.370523 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.358286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:57397 (local address 127.16.243.190:38897)
0618 22:27:00.358468 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.358472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.359365 (+ 893us) server_negotiation.cc:407] Connection header received
0618 22:27:00.360287 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.360291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.360340 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.360425 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:00.361701 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.362245 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.363466 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.363752 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.363851 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.366680 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:00.366683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:00.366685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:00.366710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:00.368238 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.368812 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.368815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.368817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.368908 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.369315 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.369319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.369321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.369564 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:00.369662 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.370196 (+ 534us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.370333 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260618 22:27:00.371313 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.358721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:38897 (local address 127.16.243.129:57397)
0618 22:27:00.359191 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.359235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.360044 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.360630 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.360637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.361032 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:00.361537 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.361550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.362354 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.362357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.363295 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.363304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.363934 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.363939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.364110 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.364764 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:00.364786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:00.366483 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:00.368394 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.368400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.368415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.368696 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.369009 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.369012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.369014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.369184 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.369665 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:00.369670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:00.369937 (+ 267us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.370097 (+ 160us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.370382 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260618 22:27:00.372641 18119 heartbeater.cc:347] Connected to a master server at 127.16.243.190:38897
I20260618 22:27:00.372915 18119 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:00.373502 18119 heartbeater.cc:511] Master 127.16.243.190:38897 requested a full tablet report, sending...
I20260618 22:27:00.375089 17928 ts_manager.cc:194] Registered new tserver with Master: 60ccd0a1ed6448bbaa4cd640d833891e (127.16.243.129:45165)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:00.376536 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:57397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:00.392967 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:00.505930 18124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:00.506234 18124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:00.506309 18124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:00.510416 18124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:00.510500 18124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:00.510624 18124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:00.516217 18124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:00.517338 18124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:00.518400 18124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:00.529565 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.529565 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:00.529692 18124 server_base.cc:1034] running on GCE node
W20260618 22:27:00.529799 18132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:00.530624 18124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:00.531452 18124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:00.532610 18124 hybrid_clock.cc:648] HybridClock initialized: now 1781821620532546 us; error 80 us; skew 500 ppm
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:00.535764 18124 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:00.536726 18124 webserver.cc:460] Webserver started at http://127.16.243.130:45851/ using document root <none> and password file <none>
I20260618 22:27:00.537294 18124 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:00.537359 18124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:00.537561 18124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:00.539691 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "eedb8d3435dd4a6db864950ef963385f"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "727f1d5c30c152bed0c67543207ba767"
server_key_iv: "928157d9d7f760a6b7b0397efe5eb7aa"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.540223 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "eedb8d3435dd4a6db864950ef963385f"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "727f1d5c30c152bed0c67543207ba767"
server_key_iv: "928157d9d7f760a6b7b0397efe5eb7aa"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.544031 18124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:00.546530 18139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.547955 18124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:00.548081 18124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "eedb8d3435dd4a6db864950ef963385f"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "727f1d5c30c152bed0c67543207ba767"
server_key_iv: "928157d9d7f760a6b7b0397efe5eb7aa"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.548190 18124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:00.569790 18124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:00.573146 18124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:00.573355 18124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:00.573979 18124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:00.574913 18124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:00.574961 18124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.575014 18124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:00.575042 18124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.583702 18124 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:39309
I20260618 22:27:00.583746 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:39309 every 8 connection(s)
I20260618 22:27:00.584619 18124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:00.589640 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18124
I20260618 22:27:00.589758 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:00.590026 17358 external_mini_cluster.cc:1428] Setting key 585537761aeb7894faec5f690a518d4d
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:00.599013 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.586494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:36347 (local address 127.16.243.190:38897)
0618 22:27:00.586656 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.586661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.587599 (+ 938us) server_negotiation.cc:407] Connection header received
0618 22:27:00.588495 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.588498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.588558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.588630 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:00.590059 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.590736 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.592303 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.592538 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.592624 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.595322 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:00.595325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:00.595334 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:00.595359 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:00.596938 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.597453 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.597457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.597459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.597538 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.597860 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.597864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.597866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.598056 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:00.598137 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.598708 (+ 571us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.598844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 22:27:00.599956 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.586957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:38897 (local address 127.16.243.130:36347)
0618 22:27:00.587417 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.587469 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.588252 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.588822 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.588850 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.589221 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:00.589885 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.589907 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.590885 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.590887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.592161 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.592174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.592692 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.592695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.592907 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.593554 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:00.593571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:00.595142 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:00.597052 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.597058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.597078 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.597352 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.597621 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.597624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.597626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.597767 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.598154 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:00.598161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:00.598427 (+ 266us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.598643 (+ 216us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.598995 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:00.601168 18253 heartbeater.cc:347] Connected to a master server at 127.16.243.190:38897
I20260618 22:27:00.601464 18253 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:00.602056 18253 heartbeater.cc:511] Master 127.16.243.190:38897 requested a full tablet report, sending...
I20260618 22:27:00.603240 17928 ts_manager.cc:194] Registered new tserver with Master: eedb8d3435dd4a6db864950ef963385f (127.16.243.130:39309)
I20260618 22:27:00.603832 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:36347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:00.617061 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:00.728472 18258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:00.728768 18258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:00.728828 18258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:00.732972 18258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:00.733048 18258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:00.733134 18258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:00.738749 18258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:38897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:00.739878 18258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:00.740942 18258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:00.750530 18264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.750586 18263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:00.750530 18266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:00.750895 18258 server_base.cc:1034] running on GCE node
I20260618 22:27:00.751322 18258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:00.752020 18258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:00.753192 18258 hybrid_clock.cc:648] HybridClock initialized: now 1781821620753167 us; error 42 us; skew 500 ppm
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:00.755981 18258 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:00.756891 18258 webserver.cc:460] Webserver started at http://127.16.243.131:46171/ using document root <none> and password file <none>
I20260618 22:27:00.757464 18258 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:00.757532 18258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:00.757727 18258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:00.759891 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "6f07fbfe46f74fef8faa51d8a7ae4087"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "9b643614aec42501d1752afee3c2b588"
server_key_iv: "e9260192ec9af50329d2a94ada153f0b"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.760488 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "6f07fbfe46f74fef8faa51d8a7ae4087"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "9b643614aec42501d1752afee3c2b588"
server_key_iv: "e9260192ec9af50329d2a94ada153f0b"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.764413 18258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260618 22:27:00.767053 18273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.768321 18258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:00.768448 18258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "6f07fbfe46f74fef8faa51d8a7ae4087"
format_stamp: "Formatted at 2026-06-18 22:27:00 on dist-test-slave-jm5g"
server_key: "9b643614aec42501d1752afee3c2b588"
server_key_iv: "e9260192ec9af50329d2a94ada153f0b"
server_key_version: "encryptionkey@0"
I20260618 22:27:00.768555 18258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:00.785071 18258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:00.788527 18258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:00.788719 18258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:00.789335 18258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:00.790184 18258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:00.790231 18258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.790283 18258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:00.790311 18258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:00.799377 18258 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:39257
I20260618 22:27:00.799415 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:39257 every 8 connection(s)
I20260618 22:27:00.800357 18258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:00.803531 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18258
I20260618 22:27:00.803658 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:00.803948 17358 external_mini_cluster.cc:1428] Setting key b14e1c3e84ee0f2bfb5f00d4c9e89fa2
Jun 18 22:27:00 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821620, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:00.814723 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.802262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50481 (local address 127.16.243.190:38897)
0618 22:27:00.802417 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.802421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.803654 (+ 1233us) server_negotiation.cc:407] Connection header received
0618 22:27:00.804802 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.804805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.804860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.804948 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:00.806377 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.806983 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.808146 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.808395 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.808498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.811065 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:00.811069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:00.811071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:00.811098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:00.812607 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.813205 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.813207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.813209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.813288 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.813595 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.813597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.813599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.813799 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:00.813878 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.814440 (+ 562us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.814585 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:27:00.815654 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.802759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:38897 (local address 127.16.243.131:50481)
0618 22:27:00.803407 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.803498 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.804577 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.805174 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.805182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.805622 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:00.806243 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.806260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.807103 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.807106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.808049 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.808058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.808530 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.808532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.808689 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.809265 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:00.809282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:00.810803 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:00.812756 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.812762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.812779 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.813103 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.813364 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:00.813366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:00.813368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:00.813525 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:00.813897 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:00.813903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:00.814171 (+ 268us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.814390 (+ 219us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.814733 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":133,"threads_started":1}
I20260618 22:27:00.817092 18387 heartbeater.cc:347] Connected to a master server at 127.16.243.190:38897
I20260618 22:27:00.817375 18387 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:00.817957 18387 heartbeater.cc:511] Master 127.16.243.190:38897 requested a full tablet report, sending...
I20260618 22:27:00.819362 17928 ts_manager.cc:194] Registered new tserver with Master: 6f07fbfe46f74fef8faa51d8a7ae4087 (127.16.243.131:39257)
I20260618 22:27:00.820096 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:50481
I20260618 22:27:00.830125 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:27:00.841377 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.833789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.16.243.190:38897)
0618 22:27:00.833937 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.833941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.834051 (+ 110us) server_negotiation.cc:407] Connection header received
0618 22:27:00.834163 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.834165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.834212 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.834322 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:00.835092 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.835615 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.836442 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.836660 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.836741 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.838228 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:00.838231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:00.838234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:00.838258 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:00.839621 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.840091 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.840094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.840095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.840187 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:00.840458 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:00.840462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:00.840463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:00.840749 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:00.840875 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.841102 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.841230 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:00.846868 17928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:00.850118 17928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:00.865578 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.859844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.16.243.129:45165)
0618 22:27:00.860236 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.860254 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.860277 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:27:00.860348 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.860353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.860549 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.861005 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:00.861156 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862240 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.863731 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.864694 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.864866 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.865022 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.865304 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.865414 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":78,"threads_started":1}
I20260618 22:27:00.865568 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.859280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45165 (local address 127.0.0.1:44472)
0618 22:27:00.859711 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.859757 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.860003 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.860682 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.860685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.860701 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:00.860858 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.860863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862384 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.862387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.863610 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.863618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.864880 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.864884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.865019 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.865113 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.865234 (+ 121us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.865379 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":99,"thread_start_us":97,"threads_started":1}
I20260618 22:27:00.865568 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.859670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:39309 (local address 127.0.0.1:53442)
0618 22:27:00.859957 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.859969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.860065 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.861182 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.861185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.861199 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:00.861291 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.861297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862246 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.862249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.863788 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.863795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.864942 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.864946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.865050 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.865112 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.865234 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.865379 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":63,"thread_start_us":94,"threads_started":1}
I20260618 22:27:00.866233 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.860265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.16.243.130:39309)
0618 22:27:00.860801 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.860812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.860834 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:27:00.860903 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.860908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.861064 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.861187 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:00.861407 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862124 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.863905 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.864818 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.864932 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.865028 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.865788 (+ 760us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.865895 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":94,"threads_started":1}
I20260618 22:27:00.866706 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.858945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:39257 (local address 127.0.0.1:36896)
0618 22:27:00.859342 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:00.859405 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:00.859543 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:00.860637 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:00.860645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:00.860670 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:00.860831 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.860838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862336 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.862339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.863785 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:00.863795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.866273 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.866277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:00.866374 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.866396 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:00.866463 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:27:00.866537 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":103,"thread_start_us":83,"threads_started":1}
I20260618 22:27:00.867172 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:00.859057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.16.243.131:39257)
0618 22:27:00.859539 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:00.859552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:00.859575 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:27:00.859740 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:00.859746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:00.860468 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:00.860637 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:00.860963 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.862201 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.865091 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:00.865963 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:00.866082 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:00.866206 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:00.866981 (+ 775us) server_negotiation.cc:299] Negotiation successful
0618 22:27:00.867068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":139,"threads_started":1}
I20260618 22:27:00.869282 18187 tablet_service.cc:1467] Processing CreateTablet for tablet ee186304251b469ab1c67318d38e830f (DEFAULT_TABLE table=test-table [id=0021de43a75f46438fef73c275cbc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:00.869335 18053 tablet_service.cc:1467] Processing CreateTablet for tablet ee186304251b469ab1c67318d38e830f (DEFAULT_TABLE table=test-table [id=0021de43a75f46438fef73c275cbc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:00.869704 18321 tablet_service.cc:1467] Processing CreateTablet for tablet ee186304251b469ab1c67318d38e830f (DEFAULT_TABLE table=test-table [id=0021de43a75f46438fef73c275cbc49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:00.870296 18187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee186304251b469ab1c67318d38e830f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:00.870661 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee186304251b469ab1c67318d38e830f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:00.870771 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee186304251b469ab1c67318d38e830f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:00.877158 18404 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Bootstrap starting.
I20260618 22:27:00.878859 18406 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Bootstrap starting.
I20260618 22:27:00.879781 18404 tablet_bootstrap.cc:654] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:00.880620 18404 log.cc:824] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:00.880738 18406 tablet_bootstrap.cc:654] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:00.881385 18405 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Bootstrap starting.
I20260618 22:27:00.881995 18406 log.cc:824] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:00.882983 18404 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: No bootstrap required, opened a new log
I20260618 22:27:00.883191 18404 ts_tablet_manager.cc:1397] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:00.883673 18405 tablet_bootstrap.cc:654] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:00.883937 18406 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: No bootstrap required, opened a new log
I20260618 22:27:00.884114 18406 ts_tablet_manager.cc:1397] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:00.884490 18405 log.cc:824] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:00.886621 18405 tablet_bootstrap.cc:492] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: No bootstrap required, opened a new log
I20260618 22:27:00.886821 18405 ts_tablet_manager.cc:1397] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:00.887037 18404 raft_consensus.cc:348] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.887156 18406 raft_consensus.cc:348] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.887338 18404 raft_consensus.cc:374] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:00.887377 18406 raft_consensus.cc:374] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:00.887425 18404 raft_consensus.cc:729] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedb8d3435dd4a6db864950ef963385f, State: Initialized, Role: FOLLOWER
I20260618 22:27:00.887454 18406 raft_consensus.cc:729] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f07fbfe46f74fef8faa51d8a7ae4087, State: Initialized, Role: FOLLOWER
I20260618 22:27:00.887884 18406 consensus_queue.cc:260] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.887930 18404 consensus_queue.cc:260] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.889025 18404 ts_tablet_manager.cc:1428] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:27:00.889241 18406 ts_tablet_manager.cc:1428] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:27:00.889428 18253 heartbeater.cc:503] Master 127.16.243.190:38897 was elected leader, sending a full tablet report...
I20260618 22:27:00.889569 18387 heartbeater.cc:503] Master 127.16.243.190:38897 was elected leader, sending a full tablet report...
I20260618 22:27:00.891213 18405 raft_consensus.cc:348] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.891717 18405 raft_consensus.cc:374] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:00.891785 18405 raft_consensus.cc:729] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ccd0a1ed6448bbaa4cd640d833891e, State: Initialized, Role: FOLLOWER
I20260618 22:27:00.892282 18405 consensus_queue.cc:260] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:00.893102 18119 heartbeater.cc:503] Master 127.16.243.190:38897 was elected leader, sending a full tablet report...
I20260618 22:27:00.893428 18405 ts_tablet_manager.cc:1428] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260618 22:27:01.051326 18388 tablet.cc:2369] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:01.085784 18254 tablet.cc:2369] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:01.100679 18410 raft_consensus.cc:482] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:01.100900 18410 raft_consensus.cc:504] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:01.102098 18410 leader_election.cc:288] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f07fbfe46f74fef8faa51d8a7ae4087 (127.16.243.131:39257), 60ccd0a1ed6448bbaa4cd640d833891e (127.16.243.129:45165)
I20260618 22:27:01.107445 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.102511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:51571 (local address 127.16.243.129:45165)
0618 22:27:01.102654 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:01.102657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:01.102960 (+ 303us) server_negotiation.cc:407] Connection header received
0618 22:27:01.103054 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:01.103057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:01.103103 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:01.103184 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:01.103603 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.104399 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106132 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.106791 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106913 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.106976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:01.107225 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:27:01.107306 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260618 22:27:01.107445 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.102493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:37655 (local address 127.16.243.131:39257)
0618 22:27:01.102640 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:01.102645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:01.102872 (+ 227us) server_negotiation.cc:407] Connection header received
0618 22:27:01.103038 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:01.103041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:01.103097 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:01.103183 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:01.103603 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.104393 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106131 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.106790 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106928 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.106976 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:01.107225 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:27:01.107306 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:27:01.107473 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.102400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45165 (local address 127.16.243.130:51571)
0618 22:27:01.102873 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:01.102884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:01.102965 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:01.103204 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:01.103207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:01.103250 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:01.103488 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:01.103497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.104522 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.104526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:01.105987 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:01.106000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.106914 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:01.107055 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.107114 (+ 59us) client_negotiation.cc:769] Sending connection context
0618 22:27:01.107184 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:27:01.107280 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":158,"spinlock_wait_cycles":4096,"thread_start_us":52,"threads_started":1}
W20260618 22:27:01.107895 18120 tablet.cc:2369] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:01.107475 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.102393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:39257 (local address 127.16.243.130:37655)
0618 22:27:01.102772 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:01.102788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:01.102919 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:01.103204 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:01.103207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:01.103249 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:01.103491 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:01.103497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.104522 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.104526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:01.105987 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:01.106000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.106914 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.106918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:01.107064 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.107103 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:27:01.107184 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:27:01.107280 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":233,"spinlock_wait_cycles":35200,"thread_start_us":85,"threads_started":1}
I20260618 22:27:01.108139 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee186304251b469ab1c67318d38e830f" candidate_uuid: "eedb8d3435dd4a6db864950ef963385f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" is_pre_election: true
I20260618 22:27:01.108446 18073 raft_consensus.cc:2454] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedb8d3435dd4a6db864950ef963385f in term 0.
I20260618 22:27:01.108664 18341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee186304251b469ab1c67318d38e830f" candidate_uuid: "eedb8d3435dd4a6db864950ef963385f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" is_pre_election: true
I20260618 22:27:01.108951 18341 raft_consensus.cc:2454] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedb8d3435dd4a6db864950ef963385f in term 0.
I20260618 22:27:01.108989 18143 leader_election.cc:302] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ccd0a1ed6448bbaa4cd640d833891e, eedb8d3435dd4a6db864950ef963385f; no voters:
I20260618 22:27:01.109313 18410 raft_consensus.cc:2781] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:01.109403 18410 raft_consensus.cc:482] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:01.109437 18410 raft_consensus.cc:3037] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:01.110502 18410 raft_consensus.cc:504] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:01.110884 18410 leader_election.cc:288] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [CANDIDATE]: Term 1 election: Requested vote from peers 6f07fbfe46f74fef8faa51d8a7ae4087 (127.16.243.131:39257), 60ccd0a1ed6448bbaa4cd640d833891e (127.16.243.129:45165)
I20260618 22:27:01.111240 18341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee186304251b469ab1c67318d38e830f" candidate_uuid: "eedb8d3435dd4a6db864950ef963385f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087"
I20260618 22:27:01.111276 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee186304251b469ab1c67318d38e830f" candidate_uuid: "eedb8d3435dd4a6db864950ef963385f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ccd0a1ed6448bbaa4cd640d833891e"
I20260618 22:27:01.111385 18073 raft_consensus.cc:3037] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:01.111371 18341 raft_consensus.cc:3037] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:01.112654 18073 raft_consensus.cc:2454] T ee186304251b469ab1c67318d38e830f P 60ccd0a1ed6448bbaa4cd640d833891e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedb8d3435dd4a6db864950ef963385f in term 1.
I20260618 22:27:01.112654 18341 raft_consensus.cc:2454] T ee186304251b469ab1c67318d38e830f P 6f07fbfe46f74fef8faa51d8a7ae4087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedb8d3435dd4a6db864950ef963385f in term 1.
I20260618 22:27:01.112998 18142 leader_election.cc:302] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f07fbfe46f74fef8faa51d8a7ae4087, eedb8d3435dd4a6db864950ef963385f; no voters:
I20260618 22:27:01.113211 18410 raft_consensus.cc:2781] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:01.113425 18410 raft_consensus.cc:686] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [term 1 LEADER]: Becoming Leader. State: Replica: eedb8d3435dd4a6db864950ef963385f, State: Running, Role: LEADER
I20260618 22:27:01.113791 18410 consensus_queue.cc:237] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } }
I20260618 22:27:01.117175 17926 catalog_manager.cc:5495] T ee186304251b469ab1c67318d38e830f P eedb8d3435dd4a6db864950ef963385f reported cstate change: term changed from 0 to 1, leader changed from <none> to eedb8d3435dd4a6db864950ef963385f (127.16.243.130). New cstate: current_term: 1 leader_uuid: "eedb8d3435dd4a6db864950ef963385f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f07fbfe46f74fef8faa51d8a7ae4087" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 39257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eedb8d3435dd4a6db864950ef963385f" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ccd0a1ed6448bbaa4cd640d833891e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45165 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:01.161172 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.151685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.16.243.129:45165)
0618 22:27:01.151854 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:01.151858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:01.151962 (+ 104us) server_negotiation.cc:407] Connection header received
0618 22:27:01.152085 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:01.152088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:01.152133 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:01.152212 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:01.153145 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.153839 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.154726 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.154966 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.155051 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.157711 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:01.157731 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:01.157746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:01.157784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:01.159568 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.160032 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.160037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.160039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.160122 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.160326 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.160328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.160329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.160576 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:01.160726 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:01.160869 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 22:27:01.160974 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:01.187723 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.178901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.16.243.129:45165)
0618 22:27:01.179036 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:01.179041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:01.179208 (+ 167us) server_negotiation.cc:407] Connection header received
0618 22:27:01.179351 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:01.179355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:01.179404 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:01.179476 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:01.180196 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.180731 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.181575 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.181809 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.181909 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.184418 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:01.184421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:01.184423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:01.184453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:01.185964 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.186402 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.186404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.186406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.186493 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.186857 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.186859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.186861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.187055 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:01.187157 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:01.187472 (+ 315us) server_negotiation.cc:299] Negotiation successful
0618 22:27:01.187580 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:01.212806 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:01.204431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.16.243.129:45165)
0618 22:27:01.204563 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:01.204567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:01.204726 (+ 159us) server_negotiation.cc:407] Connection header received
0618 22:27:01.204901 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:01.204905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:01.204951 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:01.205020 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:01.205709 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.206209 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.207161 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:01.207379 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:01.207482 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:01.209841 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:01.209845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:01.209847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:01.209874 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:01.211221 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.211671 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.211674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.211675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.211785 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:01.212017 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:01.212019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:01.212020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:01.212197 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:01.212303 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:01.212527 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:27:01.212638 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:27:01.214206 18033 tablet_service.cc:2908] Rejecting scan request for tablet ee186304251b469ab1c67318d38e830f: Uninitialized: clean time has not yet been initialized
I20260618 22:27:01.216228 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17990
I20260618 22:27:01.221190 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18124
I20260618 22:27:01.225817 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18258
I20260618 22:27:01.230297 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 17896
2026-06-18T22:27:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2400 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[18429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[18429](info): set up 2 sockets
Jun 18 22:27:01 dist-test-slave-jm5g krb5kdc[18429](info): commencing operation
krb5kdc: starting...
W20260618 22:27:02.186151 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:02.301584 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:36779
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:36779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:02.411559 18442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:02.411863 18442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:02.411931 18442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:02.416147 18442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:02.416229 18442 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:27:02.416253 18442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:02.416276 18442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:02.416301 18442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:02.416328 18442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:02.422005 18442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:36779
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:36779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:02.423189 18442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:02.424260 18442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:02.433058 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:02.433082 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:02.433148 18442 server_base.cc:1034] running on GCE node
W20260618 22:27:02.433071 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:02.433915 18442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:02.435168 18442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:02.436353 18442 hybrid_clock.cc:648] HybridClock initialized: now 1781821622436336 us; error 36 us; skew 500 ppm
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:02.443848 18442 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:02.444859 18442 webserver.cc:460] Webserver started at http://127.16.243.190:42737/ using document root <none> and password file <none>
I20260618 22:27:02.445451 18442 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:02.445520 18442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:02.445736 18442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:02.447962 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "e0c6dc32e4034a2ab05f44d1804cc992"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "503e3adee15574be703a94c6288093ea"
server_key_iv: "a3dfe9a8e1d3a78fa14c8ff21a7e8ef5"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.448518 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "e0c6dc32e4034a2ab05f44d1804cc992"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "503e3adee15574be703a94c6288093ea"
server_key_iv: "a3dfe9a8e1d3a78fa14c8ff21a7e8ef5"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.452411 18442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260618 22:27:02.454870 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:02.456248 18442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:27:02.456365 18442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "e0c6dc32e4034a2ab05f44d1804cc992"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "503e3adee15574be703a94c6288093ea"
server_key_iv: "a3dfe9a8e1d3a78fa14c8ff21a7e8ef5"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.456485 18442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:02.466918 18442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:02.470155 18442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:02.470324 18442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:02.477936 18442 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:36779
I20260618 22:27:02.477952 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:36779 every 8 connection(s)
I20260618 22:27:02.478976 18442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:02.481876 18520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:02.487926 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Bootstrap starting.
I20260618 22:27:02.487962 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18442
I20260618 22:27:02.488057 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:02.488277 17358 external_mini_cluster.cc:1428] Setting key 7a1410f4cb7f5e945a10beec02aab9c0
I20260618 22:27:02.490770 18520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:02.491750 18520 log.cc:824] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:02.494268 18520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: No bootstrap required, opened a new log
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:02.498378 18520 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } }
I20260618 22:27:02.498715 18520 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:02.498785 18520 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c6dc32e4034a2ab05f44d1804cc992, State: Initialized, Role: FOLLOWER
I20260618 22:27:02.499331 18520 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } }
I20260618 22:27:02.499511 18520 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:02.499598 18520 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:02.499711 18520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:02.501021 18520 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } }
I20260618 22:27:02.501430 18520 leader_election.cc:302] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c6dc32e4034a2ab05f44d1804cc992; no voters:
I20260618 22:27:02.501828 18520 leader_election.cc:288] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:02.501993 18525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:02.502235 18525 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [term 1 LEADER]: Becoming Leader. State: Replica: e0c6dc32e4034a2ab05f44d1804cc992, State: Running, Role: LEADER
I20260618 22:27:02.502640 18525 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } }
I20260618 22:27:02.503352 18520 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:02.504385 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } } }
I20260618 22:27:02.504508 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:02.504809 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c6dc32e4034a2ab05f44d1804cc992. Latest consensus state: current_term: 1 leader_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c6dc32e4034a2ab05f44d1804cc992" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36779 } } }
I20260618 22:27:02.504863 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.489724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.16.243.190:36779)
0618 22:27:02.490203 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:02.490224 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:02.490270 (+ 46us) server_negotiation.cc:407] Connection header received
0618 22:27:02.491260 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:02.491286 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:02.491755 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:02.492171 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:02.492984 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.494418 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.495353 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.496156 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.496345 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.498830 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:02.498838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:02.498860 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:02.498902 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:02.500966 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.501484 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.501492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.501499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.501659 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.502123 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.502127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.502129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.502729 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:02.502947 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:02.503233 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:27:02.503777 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
I20260618 22:27:02.504884 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992 [sys.catalog]: This master's current role is: LEADER
W20260618 22:27:02.509009 18540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:27:02.509105 18540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:27:02.509163 18534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:02.510020 18534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:02.516094 18534 catalog_manager.cc:1347] Generated new cluster ID: 7b7121104a1144c5abb37b388aed2267
I20260618 22:27:02.516194 18534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:02.535982 18534 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:02.536927 18534 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:02.550452 18534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Generated new TSK 0
I20260618 22:27:02.551461 18534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:02.582684 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36779
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:02.695823 18546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:02.696126 18546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:02.696172 18546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:02.700440 18546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:02.700525 18546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:02.700567 18546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:02.700600 18546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:02.706327 18546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:02.707518 18546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:02.708568 18546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:02.718668 18554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:02.718711 18551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:02.718771 18546 server_base.cc:1034] running on GCE node
W20260618 22:27:02.718995 18552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:02.719445 18546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:02.720129 18546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:02.721305 18546 hybrid_clock.cc:648] HybridClock initialized: now 1781821622721280 us; error 40 us; skew 500 ppm
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:02.724149 18546 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:02.725099 18546 webserver.cc:460] Webserver started at http://127.16.243.129:36079/ using document root <none> and password file <none>
I20260618 22:27:02.725666 18546 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:02.725735 18546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:02.725939 18546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:02.728169 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "a07f9136f909456993c3a303e5f25a0d"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "8d28d6bb39d737142ceae2419f020a1d"
server_key_iv: "6d50dd1f3168275e8631223a32ea14ca"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.728762 18546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "a07f9136f909456993c3a303e5f25a0d"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "8d28d6bb39d737142ceae2419f020a1d"
server_key_iv: "6d50dd1f3168275e8631223a32ea14ca"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.732771 18546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:27:02.735435 18561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:02.736748 18546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:02.736873 18546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "a07f9136f909456993c3a303e5f25a0d"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "8d28d6bb39d737142ceae2419f020a1d"
server_key_iv: "6d50dd1f3168275e8631223a32ea14ca"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.736986 18546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:02.753245 18546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:02.756635 18546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:02.756821 18546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:02.758358 18546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:02.758414 18546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:02.758469 18546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:02.758497 18546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:02.769059 18546 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:45979
I20260618 22:27:02.770115 18546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:02.770999 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:45979 every 8 connection(s)
I20260618 22:27:02.772588 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.759287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:47639 (local address 127.16.243.190:36779)
0618 22:27:02.759441 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:02.759447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:02.760464 (+ 1017us) server_negotiation.cc:407] Connection header received
0618 22:27:02.761734 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:02.761739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:02.761808 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:02.761919 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:02.763544 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.764148 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.765459 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.765693 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.765787 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.768685 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:02.768689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:02.768692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:02.768725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:02.770275 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.771037 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.771041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.771044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.771145 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.771537 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.771539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.771541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.771736 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:02.771814 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:02.772342 (+ 528us) server_negotiation.cc:299] Negotiation successful
0618 22:27:02.772464 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:27:02.773417 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.759700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.129:47639)
0618 22:27:02.760270 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:02.760333 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:02.761475 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:02.762132 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:02.762142 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:02.762722 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:02.763335 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.763352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.764282 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.764286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.765329 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.765340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.765890 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.765895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.766058 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.766834 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:02.766851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:02.768470 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:02.770412 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:02.770418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:02.770434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:02.770889 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:02.771241 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:02.771245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:02.771247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:02.771422 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:02.771877 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:02.771883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:02.772130 (+ 247us) client_negotiation.cc:769] Sending connection context
0618 22:27:02.772293 (+ 163us) client_negotiation.cc:240] Negotiation successful
0618 22:27:02.772573 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":152,"threads_started":1}
I20260618 22:27:02.774811 18677 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36779
I20260618 22:27:02.774964 18677 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:02.775475 18677 heartbeater.cc:511] Master 127.16.243.190:36779 requested a full tablet report, sending...
I20260618 22:27:02.777194 18472 ts_manager.cc:194] Registered new tserver with Master: a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979)
I20260618 22:27:02.778362 18472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:47639
I20260618 22:27:02.779138 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18546
I20260618 22:27:02.779253 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:02.779546 17358 external_mini_cluster.cc:1428] Setting key a702fc9113fd1d3e06c0c86bb5282037
I20260618 22:27:02.784334 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.777059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:57919 (local address 127.16.243.190:36779)
0618 22:27:02.777192 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:02.777196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:02.777427 (+ 231us) server_negotiation.cc:407] Connection header received
0618 22:27:02.777544 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:02.777548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:02.777593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:02.777656 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:02.778123 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.778718 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.779802 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.780116 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.780221 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.781466 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:02.781470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:02.781473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:02.781497 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:02.782815 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.783208 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.783212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.783214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.783291 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:02.783555 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:02.783558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:02.783560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:02.783743 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:02.783828 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:02.784077 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:27:02.784179 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:27:02.784354 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.776958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.129:57919)
0618 22:27:02.777340 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:02.777355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:02.777459 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:02.777695 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:02.777698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:02.777909 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:02.778011 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.778018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.778839 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.778843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.779669 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.779682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.780237 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.780240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.780338 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.780981 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:02.780993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:02.781324 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:02.782917 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:02.782921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:02.782923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:02.783119 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:02.783375 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:02.783378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:02.783380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:02.783473 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:02.783832 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:02.783835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:02.783925 (+ 90us) client_negotiation.cc:769] Sending connection context
0618 22:27:02.784012 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:27:02.784159 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:02.806035 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36779
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:02.918632 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:02.918958 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:02.919027 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:02.923383 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:02.923484 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:02.923547 18686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:02.923602 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:02.929280 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:02.930426 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:02.931528 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:02.941465 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:02.941468 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:02.941565 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:02.941807 18686 server_base.cc:1034] running on GCE node
I20260618 22:27:02.942260 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:02.943037 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:02.944214 18686 hybrid_clock.cc:648] HybridClock initialized: now 1781821622944189 us; error 33 us; skew 500 ppm
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:02.947129 18686 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:02.948073 18686 webserver.cc:460] Webserver started at http://127.16.243.130:35595/ using document root <none> and password file <none>
I20260618 22:27:02.948637 18686 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:02.948701 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:02.948904 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:02.951078 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "8858fb919e374541a977e839708133b3"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "631723f1e66919c64ef96e838e8216d6"
server_key_iv: "1240689de127ac7f626f28c1a00980b6"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.951639 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "8858fb919e374541a977e839708133b3"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "631723f1e66919c64ef96e838e8216d6"
server_key_iv: "1240689de127ac7f626f28c1a00980b6"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.955612 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:27:02.958355 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:02.959744 18686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:27:02.959905 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "8858fb919e374541a977e839708133b3"
format_stamp: "Formatted at 2026-06-18 22:27:02 on dist-test-slave-jm5g"
server_key: "631723f1e66919c64ef96e838e8216d6"
server_key_iv: "1240689de127ac7f626f28c1a00980b6"
server_key_version: "encryptionkey@0"
I20260618 22:27:02.960019 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:02.983494 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:02.986828 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:02.987021 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:02.988565 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:02.988619 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:02.988672 18686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:02.988699 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:02 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:02.999138 18686 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:42265
I20260618 22:27:02.999194 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:42265 every 8 connection(s)
I20260618 22:27:03.000180 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:03.002127 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.989383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55149 (local address 127.16.243.190:36779)
0618 22:27:02.989506 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:02.989510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:02.990630 (+ 1120us) server_negotiation.cc:407] Connection header received
0618 22:27:02.991690 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:02.991693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:02.991754 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:02.991832 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:02.993108 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.993632 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.995046 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.995347 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.995446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.998356 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:02.998359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:02.998361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:02.998389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:02.999932 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.000538 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.000542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.000544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.000649 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.001032 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.001036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.001038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.001239 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.001328 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.001831 (+ 503us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.001965 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:03.003044 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:02.989960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.130:55149)
0618 22:27:02.990451 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:02.990495 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:02.991445 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:02.992029 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:02.992037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:02.992395 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:02.992951 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.992967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.993765 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.993767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.994912 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:02.994923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:02.995515 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:02.995519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:02.995692 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:02.996324 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:02.996341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:02.998138 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:03.000061 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.000067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.000082 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.000432 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.000757 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.000761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.000763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.000949 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.001344 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:03.001352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:03.001596 (+ 244us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.001765 (+ 169us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.002108 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260618 22:27:03.003067 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18686
I20260618 22:27:03.003165 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:03.003446 17358 external_mini_cluster.cc:1428] Setting key 493d09dbcc4333ec64d344a9a4a83cfc
I20260618 22:27:03.004623 18817 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36779
I20260618 22:27:03.004773 18817 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:03.005338 18817 heartbeater.cc:511] Master 127.16.243.190:36779 requested a full tablet report, sending...
I20260618 22:27:03.006501 18471 ts_manager.cc:194] Registered new tserver with Master: 8858fb919e374541a977e839708133b3 (127.16.243.130:42265)
I20260618 22:27:03.007208 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:55149
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:03.018463 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.006792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.130:55343)
0618 22:27:03.010812 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.010828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.010954 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.011266 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.011268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.011456 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:03.011567 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.011572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.012680 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.012683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.013351 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.013363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.013971 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.013975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.014084 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.014653 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:03.014668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:03.015065 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:03.016772 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.016782 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.016784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.017044 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.017345 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.017348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.017350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.017441 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.017994 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:03.017996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:03.018086 (+ 90us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.018169 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.018302 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3914,"thread_start_us":111,"threads_started":1}
I20260618 22:27:03.018536 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.006864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55343 (local address 127.16.243.190:36779)
0618 22:27:03.006971 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.006975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.010953 (+ 3978us) server_negotiation.cc:407] Connection header received
0618 22:27:03.011071 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.011074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.011134 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.011240 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:03.011691 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.012513 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.013488 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.013816 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.013956 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.015223 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:03.015226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:03.015229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:03.015259 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:03.016667 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.017162 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.017165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.017166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.017253 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.017576 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.017579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.017580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.017893 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.017994 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.018247 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.018383 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:03.031997 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36779
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:03.144994 18826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:03.145298 18826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:03.145345 18826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:03.149516 18826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:03.149586 18826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:03.149628 18826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:03.149662 18826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:03.155139 18826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:03.156227 18826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:03.157310 18826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:03.167502 18832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:03.167516 18834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:03.167562 18826 server_base.cc:1034] running on GCE node
W20260618 22:27:03.167519 18831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:03.168210 18826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:03.168926 18826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:03.170094 18826 hybrid_clock.cc:648] HybridClock initialized: now 1781821623170072 us; error 36 us; skew 500 ppm
Jun 18 22:27:03 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:03.172981 18826 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:03.173969 18826 webserver.cc:460] Webserver started at http://127.16.243.131:35697/ using document root <none> and password file <none>
I20260618 22:27:03.174623 18826 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:03.174698 18826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:03.174911 18826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:03.177084 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "bd4258168db44e689b602eb9a48a7024"
format_stamp: "Formatted at 2026-06-18 22:27:03 on dist-test-slave-jm5g"
server_key: "2c09de6bca1edb31722a205d65495d2a"
server_key_iv: "707eedc8b05ac0b534c6579fd915cbcd"
server_key_version: "encryptionkey@0"
I20260618 22:27:03.177662 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "bd4258168db44e689b602eb9a48a7024"
format_stamp: "Formatted at 2026-06-18 22:27:03 on dist-test-slave-jm5g"
server_key: "2c09de6bca1edb31722a205d65495d2a"
server_key_iv: "707eedc8b05ac0b534c6579fd915cbcd"
server_key_version: "encryptionkey@0"
I20260618 22:27:03.181713 18826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:03.184648 18841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:03.186069 18826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:03.186195 18826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "bd4258168db44e689b602eb9a48a7024"
format_stamp: "Formatted at 2026-06-18 22:27:03 on dist-test-slave-jm5g"
server_key: "2c09de6bca1edb31722a205d65495d2a"
server_key_iv: "707eedc8b05ac0b534c6579fd915cbcd"
server_key_version: "encryptionkey@0"
I20260618 22:27:03.186321 18826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:03.201822 18826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:03.205410 18826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:03.205601 18826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:03.207162 18826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:03.207226 18826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:03.207283 18826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:03.207319 18826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:03 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821623, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:03.217355 18826 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:36271
I20260618 22:27:03.217444 18956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:36271 every 8 connection(s)
I20260618 22:27:03.218432 18826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:03.220603 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.207989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:55701 (local address 127.16.243.190:36779)
0618 22:27:03.208141 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.208145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.209107 (+ 962us) server_negotiation.cc:407] Connection header received
0618 22:27:03.210225 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.210230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.210294 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.210401 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:03.211872 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.212464 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.213882 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.214103 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.214193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.216969 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:03.216972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:03.216974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:03.217002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:03.218499 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.219151 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.219156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.219157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.219261 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.219628 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.219630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.219631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.219818 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.219903 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.220364 (+ 461us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.220470 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:27:03.221417 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.208450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.131:55701)
0618 22:27:03.208920 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.208974 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.209965 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.210642 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.210652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.211067 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:03.211687 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.211704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.212593 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.212596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.213739 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.213751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.214279 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.214283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.214495 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.215150 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:03.215168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:03.216770 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:03.218677 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.218682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.218694 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.219028 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.219361 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.219364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.219366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.219544 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.219911 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:03.219916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:03.220161 (+ 245us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.220322 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.220591 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260618 22:27:03.222617 18957 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36779
I20260618 22:27:03.222766 18957 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:03.223326 18957 heartbeater.cc:511] Master 127.16.243.190:36779 requested a full tablet report, sending...
I20260618 22:27:03.224501 18472 ts_manager.cc:194] Registered new tserver with Master: bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271)
I20260618 22:27:03.225196 18472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:55701
I20260618 22:27:03.228520 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 18826
I20260618 22:27:03.228649 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:03.228927 17358 external_mini_cluster.cc:1428] Setting key 0623f441e034f11b58000a774f637700
I20260618 22:27:03.231220 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:27:03.232373 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.224992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:44237 (local address 127.16.243.190:36779)
0618 22:27:03.225492 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.225496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.225509 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:03.225543 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.225546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.225585 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.225666 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:03.226250 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.226794 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.227732 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.227944 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.228020 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.229246 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:03.229250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:03.229253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:03.229281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:03.230826 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.231246 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.231249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.231250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.231357 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.231616 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.231618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.231619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.231799 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.231871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.232143 (+ 272us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.232242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260618 22:27:03.232427 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.224847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.16.243.131:44237)
0618 22:27:03.225150 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.225162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.225237 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.225718 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.225720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.225993 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:03.226136 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.226145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.226912 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.226915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.227626 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.227635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.228054 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.228056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.228126 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.228664 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:03.228684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:03.229106 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:03.230937 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.230940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.230942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.231157 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.231440 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.231443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.231445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.231539 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.231884 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:03.231887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:03.231996 (+ 109us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.232097 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.232245 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
Jun 18 22:27:03 dist-test-slave-jm5g krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:27:03 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:03.256747 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.248624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.16.243.190:36779)
0618 22:27:03.248808 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.248813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.248828 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:03.248873 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.248876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.248917 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.248992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:03.249712 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.250236 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.251221 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.251446 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.251536 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.253847 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:03.253850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:03.253852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:03.253881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:03.255188 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.255588 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.255591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.255593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.255667 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.255916 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.255919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.255921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.256108 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.256223 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.256474 (+ 251us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.256587 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260618 22:27:03.259972 18472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:03.263095 18472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:03.277271 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.272035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.16.243.131:36271)
0618 22:27:03.272460 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.272472 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.272494 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:27:03.272565 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.272570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.272785 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.272958 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.273414 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.274452 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.275872 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.276544 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.276692 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.276857 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.277001 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.277096 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":124,"threads_started":1}
I20260618 22:27:03.277264 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.271773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:36271 (local address 127.0.0.1:59762)
0618 22:27:03.272179 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.272231 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.272359 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.273130 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.273136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.273155 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.273288 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.273296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.274586 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.274590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.275728 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.275739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.276676 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.276680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.276797 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.276837 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.276962 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.277095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260618 22:27:03.278491 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.272581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42265 (local address 127.0.0.1:47896)
0618 22:27:03.273475 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.273484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.273556 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.273974 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.273977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.273996 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.274086 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.274093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.275420 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.275423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.276656 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.276667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.277915 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.277919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.278089 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.278154 (+ 65us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.278226 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.278320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":112,"thread_start_us":78,"threads_started":1}
I20260618 22:27:03.278491 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.271773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45979 (local address 127.0.0.1:51730)
0618 22:27:03.273055 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.273069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.273149 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.273657 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.273660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.273673 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.273951 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.273961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.275381 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.275384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.276694 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.276702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.277926 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.277930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.278107 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.278160 (+ 53us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.278226 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.278319 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":874,"thread_start_us":119,"threads_started":1}
I20260618 22:27:03.278543 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.272725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.16.243.130:42265)
0618 22:27:03.273366 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.273390 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.273559 (+ 169us) server_negotiation.cc:407] Connection header received
0618 22:27:03.273652 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.273657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.273824 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.273963 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.274204 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.275309 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.276793 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.277612 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.277740 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.277852 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.278291 (+ 439us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.278394 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":82,"threads_started":1}
I20260618 22:27:03.279685 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.271912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.16.243.129:45979)
0618 22:27:03.272978 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.272995 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.273152 (+ 157us) server_negotiation.cc:407] Connection header received
0618 22:27:03.273311 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.273315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.273505 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.273655 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.274071 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.275253 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.276804 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.277612 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.277729 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.277852 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.279424 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.279533 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":100,"threads_started":1}
I20260618 22:27:03.280742 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42274f606e42809a7a7729ab8a4369 (DEFAULT_TABLE table=test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.281755 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42274f606e42809a7a7729ab8a4369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.281785 18751 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42274f606e42809a7a7729ab8a4369 (DEFAULT_TABLE table=test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.282307 18611 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42274f606e42809a7a7729ab8a4369 (DEFAULT_TABLE table=test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.282891 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42274f606e42809a7a7729ab8a4369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.283270 18611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42274f606e42809a7a7729ab8a4369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.289347 18980 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Bootstrap starting.
I20260618 22:27:03.290877 18982 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Bootstrap starting.
I20260618 22:27:03.290884 18981 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Bootstrap starting.
I20260618 22:27:03.291812 18980 tablet_bootstrap.cc:654] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.292618 18980 log.cc:824] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:03.293028 18981 tablet_bootstrap.cc:654] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.293027 18982 tablet_bootstrap.cc:654] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.293846 18982 log.cc:824] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:03.293845 18981 log.cc:824] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:03.294847 18980 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: No bootstrap required, opened a new log
I20260618 22:27:03.295112 18980 ts_tablet_manager.cc:1397] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:03.295864 18981 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: No bootstrap required, opened a new log
I20260618 22:27:03.295889 18982 tablet_bootstrap.cc:492] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: No bootstrap required, opened a new log
I20260618 22:27:03.296051 18982 ts_tablet_manager.cc:1397] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:27:03.296060 18981 ts_tablet_manager.cc:1397] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:03.298918 18982 raft_consensus.cc:348] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.299166 18982 raft_consensus.cc:374] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.299213 18982 raft_consensus.cc:729] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8858fb919e374541a977e839708133b3, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.299324 18980 raft_consensus.cc:348] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.299604 18980 raft_consensus.cc:374] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.299665 18980 raft_consensus.cc:729] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4258168db44e689b602eb9a48a7024, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.299676 18982 consensus_queue.cc:260] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.300107 18980 consensus_queue.cc:260] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.300477 18817 heartbeater.cc:503] Master 127.16.243.190:36779 was elected leader, sending a full tablet report...
I20260618 22:27:03.300375 18981 raft_consensus.cc:348] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.300623 18981 raft_consensus.cc:374] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.300673 18981 raft_consensus.cc:729] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07f9136f909456993c3a303e5f25a0d, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.300758 18982 ts_tablet_manager.cc:1428] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:27:03.301133 18980 ts_tablet_manager.cc:1428] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:03.301108 18981 consensus_queue.cc:260] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.301291 18957 heartbeater.cc:503] Master 127.16.243.190:36779 was elected leader, sending a full tablet report...
I20260618 22:27:03.301749 18677 heartbeater.cc:503] Master 127.16.243.190:36779 was elected leader, sending a full tablet report...
I20260618 22:27:03.302083 18981 ts_tablet_manager.cc:1428] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260618 22:27:03.469506 18958 tablet.cc:2369] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:03.501423 18818 tablet.cc:2369] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:03.521801 18678 tablet.cc:2369] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:03.591070 18987 raft_consensus.cc:482] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:03.591272 18987 raft_consensus.cc:504] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.592433 18987 leader_election.cc:288] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979), 8858fb919e374541a977e839708133b3 (127.16.243.130:42265)
I20260618 22:27:03.596086 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.592962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:46155 (local address 127.16.243.129:45979)
0618 22:27:03.593083 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.593088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.593102 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:03.593152 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.593154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.593191 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.593254 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.593529 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.594050 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.595143 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.595540 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.595651 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.595709 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.595897 (+ 188us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.595974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:27:03.596086 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.592719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45979 (local address 127.16.243.131:46155)
0618 22:27:03.592873 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.592889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.592989 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.593294 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.593297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.593315 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.593421 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.593426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.594161 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.594164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.595013 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.595023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.595656 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.595659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.595772 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.595789 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.595868 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.595942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260618 22:27:03.596601 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.592876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:59681 (local address 127.16.243.130:42265)
0618 22:27:03.593008 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.593012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.593287 (+ 275us) server_negotiation.cc:407] Connection header received
0618 22:27:03.593409 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.593413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.593453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.593512 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.594128 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.594637 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.595616 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.596012 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.596116 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.596167 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.596423 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.596501 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:27:03.596596 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.592720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42265 (local address 127.16.243.131:59681)
0618 22:27:03.593184 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.593195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.593299 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.593542 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.593545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.593559 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.594007 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.594014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.594747 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.594751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.595490 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.595498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.596141 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.596144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.596242 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.596266 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.596384 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.596457 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":154,"thread_start_us":104,"threads_started":1}
I20260618 22:27:03.596761 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f42274f606e42809a7a7729ab8a4369" candidate_uuid: "bd4258168db44e689b602eb9a48a7024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f9136f909456993c3a303e5f25a0d" is_pre_election: true
I20260618 22:27:03.597097 18631 raft_consensus.cc:2454] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd4258168db44e689b602eb9a48a7024 in term 0.
I20260618 22:27:03.597160 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f42274f606e42809a7a7729ab8a4369" candidate_uuid: "bd4258168db44e689b602eb9a48a7024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8858fb919e374541a977e839708133b3" is_pre_election: true
I20260618 22:27:03.597414 18771 raft_consensus.cc:2454] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd4258168db44e689b602eb9a48a7024 in term 0.
I20260618 22:27:03.597716 18842 leader_election.cc:302] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07f9136f909456993c3a303e5f25a0d, bd4258168db44e689b602eb9a48a7024; no voters:
I20260618 22:27:03.598022 18987 raft_consensus.cc:2781] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:03.598109 18987 raft_consensus.cc:482] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:03.598138 18987 raft_consensus.cc:3037] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.599340 18987 raft_consensus.cc:504] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.599680 18987 leader_election.cc:288] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [CANDIDATE]: Term 1 election: Requested vote from peers a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979), 8858fb919e374541a977e839708133b3 (127.16.243.130:42265)
I20260618 22:27:03.600082 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f42274f606e42809a7a7729ab8a4369" candidate_uuid: "bd4258168db44e689b602eb9a48a7024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8858fb919e374541a977e839708133b3"
I20260618 22:27:03.600082 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f42274f606e42809a7a7729ab8a4369" candidate_uuid: "bd4258168db44e689b602eb9a48a7024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f9136f909456993c3a303e5f25a0d"
I20260618 22:27:03.600220 18771 raft_consensus.cc:3037] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.600220 18631 raft_consensus.cc:3037] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.601490 18631 raft_consensus.cc:2454] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd4258168db44e689b602eb9a48a7024 in term 1.
I20260618 22:27:03.601490 18771 raft_consensus.cc:2454] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd4258168db44e689b602eb9a48a7024 in term 1.
I20260618 22:27:03.601877 18843 leader_election.cc:302] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8858fb919e374541a977e839708133b3, bd4258168db44e689b602eb9a48a7024; no voters:
I20260618 22:27:03.602092 18987 raft_consensus.cc:2781] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:03.602334 18987 raft_consensus.cc:686] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 1 LEADER]: Becoming Leader. State: Replica: bd4258168db44e689b602eb9a48a7024, State: Running, Role: LEADER
I20260618 22:27:03.602720 18987 consensus_queue.cc:237] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } }
I20260618 22:27:03.606225 18474 catalog_manager.cc:5495] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd4258168db44e689b602eb9a48a7024 (127.16.243.131). New cstate: current_term: 1 leader_uuid: "bd4258168db44e689b602eb9a48a7024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:03.649439 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.645657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.16.243.131:36271)
0618 22:27:03.645796 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.645799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.645813 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:03.645974 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.645977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.646034 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.646120 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:03.646367 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.646918 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.647825 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.648064 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.648150 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.648509 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:03.648960 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:03.649047 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.649241 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.649319 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:03.655429 18631 raft_consensus.cc:1261] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Refusing update from remote peer bd4258168db44e689b602eb9a48a7024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:03.655429 18771 raft_consensus.cc:1261] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Refusing update from remote peer bd4258168db44e689b602eb9a48a7024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:03.656172 18990 consensus_queue.cc:1035] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:03.656365 18987 consensus_queue.cc:1035] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:03.665733 18994 mvcc.cc:205] Tried to move back new op lower bound from 7298341370485485568 to 7298341370279948288. Current Snapshot: MvccSnapshot[applied={T|T < 7298341370485485568}]
I20260618 22:27:03.676183 18472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:03.676359 18472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:03.678867 18472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Sending DeleteTablet for 3 replicas of tablet 1f42274f606e42809a7a7729ab8a4369
I20260618 22:27:03.679697 18611 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f42274f606e42809a7a7729ab8a4369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:03 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51730
I20260618 22:27:03.679905 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f42274f606e42809a7a7729ab8a4369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:03 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:47896
I20260618 22:27:03.680254 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f42274f606e42809a7a7729ab8a4369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:03 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:59762
I20260618 22:27:03.680469 19004 tablet_replica.cc:330] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: stopping tablet replica
I20260618 22:27:03.680469 18472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:27:03.680799 19004 raft_consensus.cc:2229] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260618 22:27:03.680871 18472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:03.681102 19003 tablet_replica.cc:330] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: stopping tablet replica
I20260618 22:27:03.681178 19004 raft_consensus.cc:2258] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:03.681535 19003 raft_consensus.cc:2229] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:03.681875 19003 raft_consensus.cc:2258] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:03.682804 19004 ts_tablet_manager.cc:1905] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:03.683413 19003 ts_tablet_manager.cc:1905] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:03.685990 19004 ts_tablet_manager.cc:1918] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:03.686187 19004 log.cc:1196] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/1f42274f606e42809a7a7729ab8a4369
I20260618 22:27:03.686138 18611 tablet_service.cc:1467] Processing CreateTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (DEFAULT_TABLE table=test-table [id=aa219bc7782e413ca7e454de3d7167a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.686100 19003 ts_tablet_manager.cc:1918] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:03.686393 18611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.686502 19004 ts_tablet_manager.cc:1939] T 1f42274f606e42809a7a7729ab8a4369 P 8858fb919e374541a977e839708133b3: Deleting consensus metadata
I20260618 22:27:03.686412 19003 log.cc:1196] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/1f42274f606e42809a7a7729ab8a4369
I20260618 22:27:03.686923 19003 ts_tablet_manager.cc:1939] T 1f42274f606e42809a7a7729ab8a4369 P a07f9136f909456993c3a303e5f25a0d: Deleting consensus metadata
I20260618 22:27:03.688185 18458 catalog_manager.cc:4841] TS a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979): tablet 1f42274f606e42809a7a7729ab8a4369 (table test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]) successfully deleted
I20260618 22:27:03.688347 18459 catalog_manager.cc:4841] TS 8858fb919e374541a977e839708133b3 (127.16.243.130:42265): tablet 1f42274f606e42809a7a7729ab8a4369 (table test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]) successfully deleted
I20260618 22:27:03.688510 18981 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Bootstrap starting.
I20260618 22:27:03.688977 18751 tablet_service.cc:1467] Processing CreateTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (DEFAULT_TABLE table=test-table [id=aa219bc7782e413ca7e454de3d7167a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.689213 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.689424 18981 tablet_bootstrap.cc:654] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.689862 19005 tablet_replica.cc:330] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: stopping tablet replica
I20260618 22:27:03.690249 19005 raft_consensus.cc:2229] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:03.690359 18891 tablet_service.cc:1467] Processing CreateTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (DEFAULT_TABLE table=test-table [id=aa219bc7782e413ca7e454de3d7167a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:03.690631 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.690878 19005 raft_consensus.cc:2258] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:03.691371 18982 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Bootstrap starting.
I20260618 22:27:03.692348 18982 tablet_bootstrap.cc:654] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.693019 18980 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Bootstrap starting.
I20260618 22:27:03.693230 19005 ts_tablet_manager.cc:1905] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:03.693885 18980 tablet_bootstrap.cc:654] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.695083 18980 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: No bootstrap required, opened a new log
I20260618 22:27:03.695173 18980 ts_tablet_manager.cc:1397] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:03.695277 18982 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: No bootstrap required, opened a new log
I20260618 22:27:03.695372 18982 ts_tablet_manager.cc:1397] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:27:03.695628 18980 raft_consensus.cc:348] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.695742 18980 raft_consensus.cc:374] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.695811 18980 raft_consensus.cc:729] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4258168db44e689b602eb9a48a7024, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.695775 18982 raft_consensus.cc:348] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.695868 18982 raft_consensus.cc:374] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.695900 18982 raft_consensus.cc:729] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8858fb919e374541a977e839708133b3, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.695945 18980 consensus_queue.cc:260] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.696007 18982 consensus_queue.cc:260] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.696233 18980 ts_tablet_manager.cc:1428] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:03.696233 18982 ts_tablet_manager.cc:1428] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:03.696924 19005 ts_tablet_manager.cc:1918] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:03.697065 19005 log.cc:1196] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/1f42274f606e42809a7a7729ab8a4369
I20260618 22:27:03.697419 19005 ts_tablet_manager.cc:1939] T 1f42274f606e42809a7a7729ab8a4369 P bd4258168db44e689b602eb9a48a7024: Deleting consensus metadata
I20260618 22:27:03.698586 18459 catalog_manager.cc:4841] TS bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271): tablet 1f42274f606e42809a7a7729ab8a4369 (table test-table [id=6b63ebbdc8644f86a9d70e56f304b9ef]) successfully deleted
I20260618 22:27:03.699371 18981 tablet_bootstrap.cc:492] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: No bootstrap required, opened a new log
I20260618 22:27:03.699468 18981 ts_tablet_manager.cc:1397] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260618 22:27:03.700016 18981 raft_consensus.cc:348] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.700115 18981 raft_consensus.cc:374] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.700136 18981 raft_consensus.cc:729] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07f9136f909456993c3a303e5f25a0d, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.700245 18981 consensus_queue.cc:260] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.700516 18981 ts_tablet_manager.cc:1428] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260618 22:27:03.822165 18986 raft_consensus.cc:482] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:03.822327 18986 raft_consensus.cc:504] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.823532 18986 leader_election.cc:288] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979), bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271)
I20260618 22:27:03.829026 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.823937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57323 (local address 127.16.243.129:45979)
0618 22:27:03.824074 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.824079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.824394 (+ 315us) server_negotiation.cc:407] Connection header received
0618 22:27:03.824447 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.824450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.824492 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.824563 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.824982 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.825762 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.827690 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.828314 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.828449 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.828504 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.828815 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.828909 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:27:03.829026 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.823891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:46625 (local address 127.16.243.131:36271)
0618 22:27:03.824013 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.824018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.824289 (+ 271us) server_negotiation.cc:407] Connection header received
0618 22:27:03.824409 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.824413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.824468 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.824563 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.824982 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.825761 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.827690 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.828334 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.828449 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.828504 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.828815 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.828894 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:03.829094 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.823829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:36271 (local address 127.16.243.130:46625)
0618 22:27:03.824156 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.824171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.824304 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.824623 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.824626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.824671 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.824886 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.824892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.825904 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.825909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.827518 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.827532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.828502 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.828507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.828648 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.828701 (+ 53us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.828805 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.828899 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":194,"spinlock_wait_cycles":10496,"thread_start_us":93,"threads_started":1}
I20260618 22:27:03.829092 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.823842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45979 (local address 127.16.243.130:57323)
0618 22:27:03.824277 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.824289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.824368 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.824623 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.824626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.824669 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.824857 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.824866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.825904 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.825909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.827518 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.827532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.828502 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.828507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.828658 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.828690 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.828777 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.828885 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":200,"spinlock_wait_cycles":30592,"thread_start_us":117,"threads_started":1}
I20260618 22:27:03.829640 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04a76dbbe9b4bc2b7e89d123ca9e3b1" candidate_uuid: "8858fb919e374541a977e839708133b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f9136f909456993c3a303e5f25a0d" is_pre_election: true
I20260618 22:27:03.829668 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04a76dbbe9b4bc2b7e89d123ca9e3b1" candidate_uuid: "8858fb919e374541a977e839708133b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4258168db44e689b602eb9a48a7024" is_pre_election: true
I20260618 22:27:03.829820 18631 raft_consensus.cc:2454] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8858fb919e374541a977e839708133b3 in term 0.
I20260618 22:27:03.829893 18911 raft_consensus.cc:2454] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8858fb919e374541a977e839708133b3 in term 0.
I20260618 22:27:03.830379 18702 leader_election.cc:302] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8858fb919e374541a977e839708133b3, a07f9136f909456993c3a303e5f25a0d, bd4258168db44e689b602eb9a48a7024; no voters:
I20260618 22:27:03.830680 18986 raft_consensus.cc:2781] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:03.830746 18986 raft_consensus.cc:482] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:03.830789 18986 raft_consensus.cc:3037] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.831826 18986 raft_consensus.cc:504] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.832185 18986 leader_election.cc:288] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [CANDIDATE]: Term 1 election: Requested vote from peers a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979), bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271)
I20260618 22:27:03.832590 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04a76dbbe9b4bc2b7e89d123ca9e3b1" candidate_uuid: "8858fb919e374541a977e839708133b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07f9136f909456993c3a303e5f25a0d"
I20260618 22:27:03.832707 18631 raft_consensus.cc:3037] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.833593 18631 raft_consensus.cc:2454] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8858fb919e374541a977e839708133b3 in term 1.
I20260618 22:27:03.833839 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04a76dbbe9b4bc2b7e89d123ca9e3b1" candidate_uuid: "8858fb919e374541a977e839708133b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4258168db44e689b602eb9a48a7024"
I20260618 22:27:03.833930 18702 leader_election.cc:302] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8858fb919e374541a977e839708133b3, a07f9136f909456993c3a303e5f25a0d; no voters:
I20260618 22:27:03.833958 18911 raft_consensus.cc:3037] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:03.834095 18986 raft_consensus.cc:2781] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:03.834316 18986 raft_consensus.cc:686] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8858fb919e374541a977e839708133b3, State: Running, Role: LEADER
I20260618 22:27:03.834599 18986 consensus_queue.cc:237] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.835040 18911 raft_consensus.cc:2454] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8858fb919e374541a977e839708133b3 in term 1.
I20260618 22:27:03.837152 18470 catalog_manager.cc:5495] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8858fb919e374541a977e839708133b3 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "8858fb919e374541a977e839708133b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:27:03 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821622, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:03.892771 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.884348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36779 (local address 127.0.0.1:34608)
0618 22:27:03.884659 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.884673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.884765 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.885012 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.885014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.885408 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:03.885510 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.885516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.886284 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.886286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.886893 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.886901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.887351 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.887354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.887451 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.887992 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:03.888003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:03.889469 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:03.891286 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.891296 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.891302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.891553 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.891829 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:03.891831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:03.891833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:03.891913 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:03.892246 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:03.892251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:03.892359 (+ 108us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.892445 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.892584 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":28,"thread_start_us":121,"threads_started":1}
I20260618 22:27:03.892778 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.884428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.16.243.190:36779)
0618 22:27:03.884713 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.884747 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.884765 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:27:03.884856 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.884858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.884903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.884987 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:03.885624 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.886189 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.887026 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.887253 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.887341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.889607 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:03.889610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:03.889611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:03.889631 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:03.891183 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.891649 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.891651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.891653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.891734 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:03.891984 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:03.891987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:03.891988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:03.892168 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:03.892241 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.892507 (+ 266us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.892640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260618 22:27:03.896592 18470 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:34608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:27:03.897346 18470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:03.902470 18611 tablet_service.cc:1467] Processing CreateTablet for tablet 156cadf8680a42f5a737a20ffcf81b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41dddecbb2004f0cb9a27d4f5fcac9be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:03.902483 18751 tablet_service.cc:1467] Processing CreateTablet for tablet 156cadf8680a42f5a737a20ffcf81b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41dddecbb2004f0cb9a27d4f5fcac9be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:03.902483 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 156cadf8680a42f5a737a20ffcf81b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41dddecbb2004f0cb9a27d4f5fcac9be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:03.902909 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156cadf8680a42f5a737a20ffcf81b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.902908 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156cadf8680a42f5a737a20ffcf81b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.902916 18611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156cadf8680a42f5a737a20ffcf81b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:03.905578 18980 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: Bootstrap starting.
I20260618 22:27:03.905579 18981 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: Bootstrap starting.
I20260618 22:27:03.905963 18982 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: Bootstrap starting.
I20260618 22:27:03.906503 18980 tablet_bootstrap.cc:654] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.906518 18981 tablet_bootstrap.cc:654] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.906872 18982 tablet_bootstrap.cc:654] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:03.907754 18980 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: No bootstrap required, opened a new log
I20260618 22:27:03.907837 18980 ts_tablet_manager.cc:1397] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:03.908052 18982 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: No bootstrap required, opened a new log
I20260618 22:27:03.908125 18981 tablet_bootstrap.cc:492] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: No bootstrap required, opened a new log
I20260618 22:27:03.908128 18982 ts_tablet_manager.cc:1397] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:03.908191 18981 ts_tablet_manager.cc:1397] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:03.908157 18980 raft_consensus.cc:348] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.908243 18980 raft_consensus.cc:374] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.908273 18980 raft_consensus.cc:729] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4258168db44e689b602eb9a48a7024, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.908380 18980 consensus_queue.cc:260] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.908624 18980 ts_tablet_manager.cc:1428] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:03.908555 18982 raft_consensus.cc:348] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.908579 18981 raft_consensus.cc:348] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.908663 18982 raft_consensus.cc:374] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.908670 18981 raft_consensus.cc:374] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:03.908605 18998 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:03.908695 18982 raft_consensus.cc:729] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8858fb919e374541a977e839708133b3, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.908699 18981 raft_consensus.cc:729] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07f9136f909456993c3a303e5f25a0d, State: Initialized, Role: FOLLOWER
I20260618 22:27:03.908730 18998 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:03.908815 18981 consensus_queue.cc:260] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.908815 18982 consensus_queue.cc:260] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.909046 18988 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:03.909174 18988 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:03.909089 18981 ts_tablet_manager.cc:1428] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:03.909076 18982 ts_tablet_manager.cc:1428] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:03.909360 18986 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:03.909483 18986 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:03.992553 18988 raft_consensus.cc:482] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:03.992695 18988 raft_consensus.cc:504] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:03.993737 18988 leader_election.cc:288] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8858fb919e374541a977e839708133b3 (127.16.243.130:42265), bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271)
I20260618 22:27:03.997815 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.994104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:54343 (local address 127.16.243.130:42265)
0618 22:27:03.994253 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.994259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.994435 (+ 176us) server_negotiation.cc:407] Connection header received
0618 22:27:03.994589 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.994592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.994642 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.994722 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.995038 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.995619 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.996857 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.997245 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.997354 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.997408 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.997634 (+ 226us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.997704 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:27:03.997815 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.994531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:50521 (local address 127.16.243.131:36271)
0618 22:27:03.994676 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:03.994681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:03.994764 (+ 83us) server_negotiation.cc:407] Connection header received
0618 22:27:03.994848 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:03.994851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:03.994924 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:03.995009 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:03.995251 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.995755 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.996853 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.997246 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.997353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.997408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:03.997621 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 22:27:03.997696 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 22:27:03.997830 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.994011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42265 (local address 127.16.243.129:54343)
0618 22:27:03.994331 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.994343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.994462 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.994765 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.994768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.994792 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.994934 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.994941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.995730 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.995733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.996730 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.996740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.997368 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.997372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.997492 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.997533 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.997600 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.997677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":65,"thread_start_us":140,"threads_started":1}
I20260618 22:27:03.997834 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:03.994422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:36271 (local address 127.16.243.129:50521)
0618 22:27:03.994680 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:03.994692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:03.994772 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:03.995024 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:03.995027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:03.995043 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:03.995152 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.995159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.995877 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.995880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.996725 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:03.996735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:03.997359 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:03.997362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:03.997468 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:03.997523 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 22:27:03.997584 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 22:27:03.997666 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":54,"thread_start_us":113,"threads_started":1}
I20260618 22:27:03.998301 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156cadf8680a42f5a737a20ffcf81b75" candidate_uuid: "a07f9136f909456993c3a303e5f25a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8858fb919e374541a977e839708133b3" is_pre_election: true
I20260618 22:27:03.998469 18771 raft_consensus.cc:2454] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07f9136f909456993c3a303e5f25a0d in term 0.
I20260618 22:27:03.998477 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156cadf8680a42f5a737a20ffcf81b75" candidate_uuid: "a07f9136f909456993c3a303e5f25a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4258168db44e689b602eb9a48a7024" is_pre_election: true
I20260618 22:27:03.998620 18911 raft_consensus.cc:2454] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07f9136f909456993c3a303e5f25a0d in term 0.
I20260618 22:27:03.998869 18563 leader_election.cc:302] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8858fb919e374541a977e839708133b3, a07f9136f909456993c3a303e5f25a0d; no voters:
I20260618 22:27:03.999383 18988 raft_consensus.cc:2781] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:03.999467 18988 raft_consensus.cc:482] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:03.999512 18988 raft_consensus.cc:3037] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:04.000725 18988 raft_consensus.cc:504] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:04.001096 18988 leader_election.cc:288] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [CANDIDATE]: Term 1 election: Requested vote from peers 8858fb919e374541a977e839708133b3 (127.16.243.130:42265), bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271)
I20260618 22:27:04.001564 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156cadf8680a42f5a737a20ffcf81b75" candidate_uuid: "a07f9136f909456993c3a303e5f25a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4258168db44e689b602eb9a48a7024"
I20260618 22:27:04.001688 18911 raft_consensus.cc:3037] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:04.002836 18911 raft_consensus.cc:2454] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07f9136f909456993c3a303e5f25a0d in term 1.
I20260618 22:27:04.003134 18563 leader_election.cc:302] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07f9136f909456993c3a303e5f25a0d, bd4258168db44e689b602eb9a48a7024; no voters:
I20260618 22:27:04.003260 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156cadf8680a42f5a737a20ffcf81b75" candidate_uuid: "a07f9136f909456993c3a303e5f25a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8858fb919e374541a977e839708133b3"
I20260618 22:27:04.003383 18771 raft_consensus.cc:3037] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:04.004498 18771 raft_consensus.cc:2454] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07f9136f909456993c3a303e5f25a0d in term 1.
I20260618 22:27:04.004703 18988 raft_consensus.cc:2781] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:04.004920 18988 raft_consensus.cc:686] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [term 1 LEADER]: Becoming Leader. State: Replica: a07f9136f909456993c3a303e5f25a0d, State: Running, Role: LEADER
I20260618 22:27:04.005160 18988 consensus_queue.cc:237] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } }
I20260618 22:27:04.007467 19025 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: TxnStatusTablet state changed. Reason: New leader a07f9136f909456993c3a303e5f25a0d. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.007589 19025 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:04.007894 19027 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:27:04.008033 19027 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:27:04.007937 18471 catalog_manager.cc:5495] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to a07f9136f909456993c3a303e5f25a0d (127.16.243.129). New cstate: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:04.039436 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.035881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45979 (local address 127.0.0.1:51732)
0618 22:27:04.036026 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:04.036040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:04.036147 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:04.036443 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:04.036446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:04.036453 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:04.036555 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.036560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.037254 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.037257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.037847 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.037855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.038271 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.038275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.038368 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.038414 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.039136 (+ 722us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.039151 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 22:27:04.039215 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:27:04.039304 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:27:04.039433 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.036010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.16.243.129:45979)
0618 22:27:04.036126 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.036131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.036144 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:04.036274 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.036277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.036332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.036406 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:04.036650 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.037149 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.037965 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.038171 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.038255 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.038591 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.039041 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.039144 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.039224 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.039280 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:27:04.042464 18771 raft_consensus.cc:1261] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Refusing update from remote peer a07f9136f909456993c3a303e5f25a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:04.042722 18911 raft_consensus.cc:1261] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Refusing update from remote peer a07f9136f909456993c3a303e5f25a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:04.042996 18988 consensus_queue.cc:1035] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:04.043233 19025 consensus_queue.cc:1035] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:04.044912 18998 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: TxnStatusTablet state changed. Reason: New leader a07f9136f909456993c3a303e5f25a0d. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.044982 18986 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: TxnStatusTablet state changed. Reason: New leader a07f9136f909456993c3a303e5f25a0d. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.045043 18998 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:04.045071 18986 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:04.046644 19025 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.046756 19025 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:04.046875 18995 mvcc.cc:205] Tried to move back new op lower bound from 7298341372072677376 to 7298341371928010752. Current Snapshot: MvccSnapshot[applied={T|T < 7298341372072677376}]
I20260618 22:27:04.047186 18986 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.047281 18986 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P 8858fb919e374541a977e839708133b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:04.049943 18988 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.050067 18988 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P a07f9136f909456993c3a303e5f25a0d: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:04.051213 18998 tablet_replica.cc:439] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a07f9136f909456993c3a303e5f25a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 } } peers { permanent_uuid: "8858fb919e374541a977e839708133b3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42265 } } peers { permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 } } }
I20260618 22:27:04.051333 18998 tablet_replica.cc:442] T 156cadf8680a42f5a737a20ffcf81b75 P bd4258168db44e689b602eb9a48a7024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:04.057219 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.053196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.16.243.130:42265)
0618 22:27:04.053301 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.053306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.053317 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:27:04.053383 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.053385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.053438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.053530 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:04.053790 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.054501 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.055673 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.055885 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.055959 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.056379 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.056837 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.056923 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.057026 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.057078 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:27:04.068213 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.064942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59233 (local address 127.16.243.129:45979)
0618 22:27:04.065067 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.065071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.065226 (+ 155us) server_negotiation.cc:407] Connection header received
0618 22:27:04.065376 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.065379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.065425 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.065486 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:04.065734 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.066251 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.067190 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.067412 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.067499 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.067787 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.067861 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.067917 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.068014 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.068064 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:04.068226 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.064789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45979 (local address 127.16.243.130:59233)
0618 22:27:04.065134 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:04.065147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:04.065263 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:04.065511 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:04.065514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:04.065522 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:04.065627 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.065638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.066359 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.066362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.067068 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.067078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.067518 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.067521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.067610 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.067655 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.067940 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.067946 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:27:04.068004 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 22:27:04.068079 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260618 22:27:04.075730 18631 raft_consensus.cc:1261] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Refusing update from remote peer 8858fb919e374541a977e839708133b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:04.075730 18911 raft_consensus.cc:1261] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Refusing update from remote peer 8858fb919e374541a977e839708133b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:04.076237 18986 consensus_queue.cc:1035] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4258168db44e689b602eb9a48a7024" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:04.076455 19011 consensus_queue.cc:1035] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07f9136f909456993c3a303e5f25a0d" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:04.082327 18993 mvcc.cc:205] Tried to move back new op lower bound from 7298341372210159616 to 7298341371229069312. Current Snapshot: MvccSnapshot[applied={T|T < 7298341372210159616}]
I20260618 22:27:04.098719 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.094696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:49815 (local address 127.16.243.130:42265)
0618 22:27:04.094807 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.094812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.095266 (+ 454us) server_negotiation.cc:407] Connection header received
0618 22:27:04.095491 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.095494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.095538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.095596 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:04.095835 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.096476 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.097491 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.097790 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.097885 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.098249 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.098333 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.098369 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.098506 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.098579 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:27:04.098730 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.094886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42265 (local address 127.16.243.129:49815)
0618 22:27:04.095184 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:04.095198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:04.095379 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:04.095617 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:04.095620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:04.095628 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:04.095733 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.095739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.096580 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.096584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.097378 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:04.097387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.097961 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.097964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:04.098046 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.098102 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:04.098417 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:04.098422 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:04.098472 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:27:04.098579 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260618 22:27:04.125965 18471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:04.126137 18471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:04.128579 18471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0c6dc32e4034a2ab05f44d1804cc992: Sending DeleteTablet for 3 replicas of tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1
I20260618 22:27:04.129495 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:04 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:59762
I20260618 22:27:04.129539 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:04 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:47896
I20260618 22:27:04.129699 19005 tablet_replica.cc:330] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: stopping tablet replica
I20260618 22:27:04.129819 19005 raft_consensus.cc:2229] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:04.129900 18611 tablet_service.cc:1514] Processing DeleteTablet for tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:04 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51730
I20260618 22:27:04.129940 19005 raft_consensus.cc:2258] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:04.130076 19003 tablet_replica.cc:330] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: stopping tablet replica
I20260618 22:27:04.130100 19004 tablet_replica.cc:330] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: stopping tablet replica
I20260618 22:27:04.130174 19003 raft_consensus.cc:2229] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:04.130198 19004 raft_consensus.cc:2229] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:04.130285 19003 raft_consensus.cc:2258] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:04.130497 19004 raft_consensus.cc:2258] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:04.131386 19003 ts_tablet_manager.cc:1905] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:04.131546 19005 ts_tablet_manager.cc:1905] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:04.134359 19003 ts_tablet_manager.cc:1918] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:04.134366 19005 ts_tablet_manager.cc:1918] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:04.134434 19003 log.cc:1196] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/b04a76dbbe9b4bc2b7e89d123ca9e3b1
I20260618 22:27:04.134440 19005 log.cc:1196] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/b04a76dbbe9b4bc2b7e89d123ca9e3b1
I20260618 22:27:04.134716 19005 ts_tablet_manager.cc:1939] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P bd4258168db44e689b602eb9a48a7024: Deleting consensus metadata
I20260618 22:27:04.134933 19003 ts_tablet_manager.cc:1939] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P a07f9136f909456993c3a303e5f25a0d: Deleting consensus metadata
I20260618 22:27:04.135572 18459 catalog_manager.cc:4841] TS bd4258168db44e689b602eb9a48a7024 (127.16.243.131:36271): tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (table test-table [id=aa219bc7782e413ca7e454de3d7167a1]) successfully deleted
I20260618 22:27:04.135771 18458 catalog_manager.cc:4841] TS a07f9136f909456993c3a303e5f25a0d (127.16.243.129:45979): tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (table test-table [id=aa219bc7782e413ca7e454de3d7167a1]) successfully deleted
I20260618 22:27:04.137151 19004 ts_tablet_manager.cc:1905] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:04.140368 19004 ts_tablet_manager.cc:1918] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:04.140511 19004 log.cc:1196] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/b04a76dbbe9b4bc2b7e89d123ca9e3b1
I20260618 22:27:04.140830 19004 ts_tablet_manager.cc:1939] T b04a76dbbe9b4bc2b7e89d123ca9e3b1 P 8858fb919e374541a977e839708133b3: Deleting consensus metadata
I20260618 22:27:04.141815 18459 catalog_manager.cc:4841] TS 8858fb919e374541a977e839708133b3 (127.16.243.130:42265): tablet b04a76dbbe9b4bc2b7e89d123ca9e3b1 (table test-table [id=aa219bc7782e413ca7e454de3d7167a1]) successfully deleted
Jun 18 22:27:04 dist-test-slave-jm5g krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:04.150861 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.137919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.16.243.129:45979)
0618 22:27:04.138032 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.138037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.138600 (+ 563us) server_negotiation.cc:407] Connection header received
0618 22:27:04.138781 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.138785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.138830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.138879 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:04.139782 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.140571 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.142062 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.142410 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.142479 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.146459 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:04.146469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:04.146487 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:04.146521 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:04.148686 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:04.149197 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:04.149204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:04.149206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:04.149310 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:04.149611 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:04.149614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:04.149616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:04.149952 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:04.150160 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.150369 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.150490 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260618 22:27:04.151295 18665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51736
I20260618 22:27:04.162456 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:04.153109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.16.243.190:36779)
0618 22:27:04.153227 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:04.153232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:04.153580 (+ 348us) server_negotiation.cc:407] Connection header received
0618 22:27:04.153760 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:04.153764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:04.153817 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:04.153866 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:04.154779 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.155592 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.156706 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:04.157047 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:04.157116 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:04.158983 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:04.158987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:04.158989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:04.159018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:04.160687 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:04.161196 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:04.161199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:04.161201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:04.161309 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:04.161577 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:04.161580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:04.161582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:04.161792 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:04.161921 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:04.162145 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:27:04.162278 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260618 22:27:04.162866 18471 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612
I20260618 22:27:04.164785 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18546
I20260618 22:27:04.171221 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18686
I20260618 22:27:04.176148 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18826
I20260618 22:27:04.181354 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 18442
2026-06-18T22:27:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2951 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:04 dist-test-slave-jm5g krb5kdc[19049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:04 dist-test-slave-jm5g krb5kdc[19049](info): set up 2 sockets
Jun 18 22:27:04 dist-test-slave-jm5g krb5kdc[19049](info): commencing operation
krb5kdc: starting...
W20260618 22:27:05.150135 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:05.267203 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41939
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:05.375968 19062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:05.376250 19062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:05.376291 19062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:05.380474 19062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:05.380575 19062 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:27:05.380594 19062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:05.380611 19062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:05.380626 19062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:05.380642 19062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:05.387820 19062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:05.388950 19062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:05.390219 19062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:05.399128 19070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:05.399145 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:05.399120 19062 server_base.cc:1034] running on GCE node
W20260618 22:27:05.399147 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:05.399907 19062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:05.400993 19062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:05.402170 19062 hybrid_clock.cc:648] HybridClock initialized: now 1781821625402142 us; error 39 us; skew 500 ppm
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:05.404896 19062 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:05.405786 19062 webserver.cc:460] Webserver started at http://127.16.243.190:34721/ using document root <none> and password file <none>
I20260618 22:27:05.406322 19062 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:05.406387 19062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:05.406611 19062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:05.408655 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "36d1cfea21ff4422a90803d113805db3"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "a9debdf7156879443558fd106b8e76d6"
server_key_iv: "6ae3f7cfafe2512d2637c9cb12e369e3"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.409206 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "36d1cfea21ff4422a90803d113805db3"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "a9debdf7156879443558fd106b8e76d6"
server_key_iv: "6ae3f7cfafe2512d2637c9cb12e369e3"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.413247 19062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260618 22:27:05.415823 19077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:05.417187 19062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:27:05.417306 19062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "36d1cfea21ff4422a90803d113805db3"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "a9debdf7156879443558fd106b8e76d6"
server_key_iv: "6ae3f7cfafe2512d2637c9cb12e369e3"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.417419 19062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:05.444239 19062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:05.447495 19062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:05.447685 19062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:05.455438 19062 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41939
I20260618 22:27:05.455447 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41939 every 8 connection(s)
I20260618 22:27:05.456413 19062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:05.459300 19140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:05.463881 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19062
I20260618 22:27:05.463994 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:05.464253 17358 external_mini_cluster.cc:1428] Setting key 83f497dd3f42536e1f72d73a41a45cfc
I20260618 22:27:05.465337 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Bootstrap starting.
I20260618 22:27:05.467737 19140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:05.468662 19140 log.cc:824] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:05.471089 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: No bootstrap required, opened a new log
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:05.475368 19140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } }
I20260618 22:27:05.475623 19140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:05.475683 19140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d1cfea21ff4422a90803d113805db3, State: Initialized, Role: FOLLOWER
I20260618 22:27:05.476284 19140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } }
I20260618 22:27:05.476475 19140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:05.476549 19140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:05.476652 19140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:05.477789 19140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } }
I20260618 22:27:05.478185 19140 leader_election.cc:302] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d1cfea21ff4422a90803d113805db3; no voters:
I20260618 22:27:05.478592 19140 leader_election.cc:288] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:05.478714 19145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:05.478955 19145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [term 1 LEADER]: Becoming Leader. State: Replica: 36d1cfea21ff4422a90803d113805db3, State: Running, Role: LEADER
I20260618 22:27:05.479347 19145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } }
I20260618 22:27:05.479835 19140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:05.480568 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.465633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.16.243.190:41939)
0618 22:27:05.466078 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:05.466095 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:05.466130 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:27:05.466860 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:05.466879 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:05.467232 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:05.467551 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:05.468549 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.469824 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.471043 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.471582 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.471724 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.474721 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:05.474726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:05.474741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:05.474772 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:05.476760 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.477214 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.477220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.477226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.477389 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.477633 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.477636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.477638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.478189 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:05.478506 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:05.479242 (+ 736us) server_negotiation.cc:299] Negotiation successful
0618 22:27:05.479579 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":137,"threads_started":1}
I20260618 22:27:05.480607 19145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d1cfea21ff4422a90803d113805db3. Latest consensus state: current_term: 1 leader_uuid: "36d1cfea21ff4422a90803d113805db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } } }
I20260618 22:27:05.480844 19145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:05.481004 19146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d1cfea21ff4422a90803d113805db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d1cfea21ff4422a90803d113805db3" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41939 } } }
I20260618 22:27:05.481099 19146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:05.481249 19154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:05.484433 19154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:05.490687 19154 catalog_manager.cc:1347] Generated new cluster ID: 0518700bd7ef450fafc319c39fd94f29
I20260618 22:27:05.490760 19154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:05.499410 19154 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:05.500381 19154 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:05.506570 19154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Generated new TSK 0
I20260618 22:27:05.507503 19154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:05.534204 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:05.651818 19166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:05.652132 19166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:05.652197 19166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:05.656291 19166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:05.656369 19166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:05.656425 19166 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:27:05.656445 19166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:05.656497 19166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:05.661978 19166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:05.663115 19166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:05.664177 19166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:05.673560 19172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:05.673560 19171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:05.673588 19174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:05.673642 19166 server_base.cc:1034] running on GCE node
I20260618 22:27:05.674201 19166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:05.674968 19166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:05.676137 19166 hybrid_clock.cc:648] HybridClock initialized: now 1781821625676110 us; error 44 us; skew 500 ppm
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:05.678903 19166 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:05.679792 19166 webserver.cc:460] Webserver started at http://127.16.243.129:39133/ using document root <none> and password file <none>
I20260618 22:27:05.680418 19166 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:05.680478 19166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:05.680647 19166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:05.682888 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "dafeff7887dca066d772181af50eba8a"
server_key_iv: "8ef558e5a2b33fed0232cb8817157df7"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.683423 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "dafeff7887dca066d772181af50eba8a"
server_key_iv: "8ef558e5a2b33fed0232cb8817157df7"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.687251 19166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:27:05.689900 19181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:05.691340 19166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:05.691455 19166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "dafeff7887dca066d772181af50eba8a"
server_key_iv: "8ef558e5a2b33fed0232cb8817157df7"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.691592 19166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:05.705247 19166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:05.708115 19166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:05.708292 19166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:05.709703 19166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:05.709750 19166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:05.709789 19166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:05.709817 19166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:05.719700 19166 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:43475
I20260618 22:27:05.719764 19296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:43475 every 8 connection(s)
I20260618 22:27:05.720777 19166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:05.723469 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19166
I20260618 22:27:05.723613 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:05.723790 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.710610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:41817 (local address 127.16.243.190:41939)
0618 22:27:05.710725 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:05.710729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:05.711835 (+ 1106us) server_negotiation.cc:407] Connection header received
0618 22:27:05.712943 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:05.712947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:05.713020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:05.713116 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:05.714572 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.715392 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.716836 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.717114 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.717210 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.720106 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:05.720110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:05.720112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:05.720143 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:05.721794 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.722256 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.722258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.722259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.722331 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.722674 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.722677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.722678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.722861 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:05.722938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:05.723483 (+ 545us) server_negotiation.cc:299] Negotiation successful
0618 22:27:05.723636 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:27:05.723894 17358 external_mini_cluster.cc:1428] Setting key f0d4d552adf68a4cfd583230df2490a0
I20260618 22:27:05.724653 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.711195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.129:41817)
0618 22:27:05.711685 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:05.711728 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:05.712696 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:05.713312 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:05.713322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:05.713719 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:05.714378 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.714399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.715524 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.715528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.716727 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.716739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.717263 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.717268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.717447 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.718063 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:05.718086 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:05.719927 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:05.721901 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.721907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.721922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.722175 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.722403 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.722405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.722406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.722542 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.722983 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:05.722988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:05.723224 (+ 236us) client_negotiation.cc:769] Sending connection context
0618 22:27:05.723393 (+ 169us) client_negotiation.cc:240] Negotiation successful
0618 22:27:05.723718 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260618 22:27:05.725944 19297 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41939
I20260618 22:27:05.726099 19297 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:05.726675 19297 heartbeater.cc:511] Master 127.16.243.190:41939 requested a full tablet report, sending...
I20260618 22:27:05.728163 19093 ts_manager.cc:194] Registered new tserver with Master: 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475)
I20260618 22:27:05.729305 19093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:41817
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:05.735872 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.728113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.129:33485)
0618 22:27:05.729327 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:05.729340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:05.729434 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:05.729669 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:05.729671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:05.729884 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:05.730000 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.730006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.730784 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.730787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.731432 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.731449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.731869 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.731873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.731959 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.732509 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:05.732519 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:05.732845 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:05.734373 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.734376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.734378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.734609 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.734874 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.734877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.734879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.734960 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.735398 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:05.735400 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:05.735483 (+ 83us) client_negotiation.cc:769] Sending connection context
0618 22:27:05.735567 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:27:05.735682 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":91,"threads_started":1}
I20260618 22:27:05.735991 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.728224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:33485 (local address 127.16.243.190:41939)
0618 22:27:05.728334 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:05.728339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:05.729428 (+ 1089us) server_negotiation.cc:407] Connection header received
0618 22:27:05.729533 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:05.729536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:05.729579 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:05.729642 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:05.730108 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.730672 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.731544 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.731768 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.731853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.732985 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:05.732989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:05.732992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:05.733014 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:05.734269 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.734714 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.734717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.734719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.734794 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.735068 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.735072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.735074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.735313 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:05.735404 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:05.735676 (+ 272us) server_negotiation.cc:299] Negotiation successful
0618 22:27:05.735830 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:05.751299 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:05.860970 19306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:05.861341 19306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:05.861418 19306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:05.865721 19306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:05.865811 19306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:05.865870 19306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:27:05.865891 19306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:05.865944 19306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:05.871690 19306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:05.872792 19306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:05.873821 19306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:05.883782 19312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:05.883785 19311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:05.883857 19306 server_base.cc:1034] running on GCE node
W20260618 22:27:05.883785 19314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:05.884498 19306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:05.885187 19306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:05.886368 19306 hybrid_clock.cc:648] HybridClock initialized: now 1781821625886344 us; error 35 us; skew 500 ppm
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:05.889148 19306 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:05.890084 19306 webserver.cc:460] Webserver started at http://127.16.243.130:35967/ using document root <none> and password file <none>
I20260618 22:27:05.890702 19306 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:05.890777 19306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:05.890992 19306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:05.893086 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "4c5084f4dcb74ae7a164f39061a8870d"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "7eae48c080662a6cd70ee6ff78bcf499"
server_key_iv: "7fffe684ceb17777eb3d8d51ba54efe8"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.893632 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "4c5084f4dcb74ae7a164f39061a8870d"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "7eae48c080662a6cd70ee6ff78bcf499"
server_key_iv: "7fffe684ceb17777eb3d8d51ba54efe8"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.897729 19306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:05.900420 19321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:05.901773 19306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:05.901903 19306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "4c5084f4dcb74ae7a164f39061a8870d"
format_stamp: "Formatted at 2026-06-18 22:27:05 on dist-test-slave-jm5g"
server_key: "7eae48c080662a6cd70ee6ff78bcf499"
server_key_iv: "7fffe684ceb17777eb3d8d51ba54efe8"
server_key_version: "encryptionkey@0"
I20260618 22:27:05.902029 19306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:05.914827 19306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:05.918114 19306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:05.918305 19306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:05.919872 19306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:05.919930 19306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:05.919994 19306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:05.920024 19306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:05 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:05.930292 19306 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:42959
I20260618 22:27:05.930395 19436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:42959 every 8 connection(s)
I20260618 22:27:05.931277 19306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:05.934067 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.920716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:48779 (local address 127.16.243.190:41939)
0618 22:27:05.920833 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:05.920837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:05.921994 (+ 1157us) server_negotiation.cc:407] Connection header received
0618 22:27:05.923100 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:05.923104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:05.923163 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:05.923244 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:05.924974 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.925781 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.927283 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.927589 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.927713 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.930647 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:05.930651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:05.930654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:05.930683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:05.932142 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.932736 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.932740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.932742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.932842 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.933147 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.933150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.933151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.933334 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:05.933405 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:05.933835 (+ 430us) server_negotiation.cc:299] Negotiation successful
0618 22:27:05.933946 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:27:05.935021 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.921316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.130:48779)
0618 22:27:05.921825 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:05.921886 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:05.922834 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:05.923474 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:05.923485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:05.924000 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:05.924744 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.924767 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.925911 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.925915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.927134 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.927146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.927772 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.927776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.927996 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.928643 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:05.928666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:05.930401 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:05.932256 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.932262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.932279 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.932605 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.932933 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.932935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.932936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.933069 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.933409 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:05.933414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:05.933645 (+ 231us) client_negotiation.cc:769] Sending connection context
0618 22:27:05.933793 (+ 148us) client_negotiation.cc:240] Negotiation successful
0618 22:27:05.934101 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":137,"threads_started":1}
I20260618 22:27:05.936307 19437 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41939
I20260618 22:27:05.936452 19437 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:05.936978 19437 heartbeater.cc:511] Master 127.16.243.190:41939 requested a full tablet report, sending...
I20260618 22:27:05.937565 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19306
I20260618 22:27:05.937686 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:05.937994 17358 external_mini_cluster.cc:1428] Setting key 548462eaaa4c0046fd24ccd55296deb3
I20260618 22:27:05.938028 19094 ts_manager.cc:194] Registered new tserver with Master: 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:05.938781 19094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:48779
I20260618 22:27:05.946062 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.938523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:54533 (local address 127.16.243.190:41939)
0618 22:27:05.938661 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:05.938665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:05.938958 (+ 293us) server_negotiation.cc:407] Connection header received
0618 22:27:05.939055 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:05.939058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:05.939102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:05.939169 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:05.939796 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.940525 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.941480 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.941779 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.941869 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.943121 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:05.943125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:05.943128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:05.943155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:05.944472 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.944939 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.944941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.944942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.945011 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:05.945270 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:05.945272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:05.945273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:05.945480 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:05.945553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:05.945833 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 22:27:05.945936 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:27:05.946148 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:05.938428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.130:54533)
0618 22:27:05.938857 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:05.938871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:05.938972 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:05.939272 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:05.939276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:05.939550 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:05.939668 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.939675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.940678 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.940681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.941352 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:05.941360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:05.941908 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:05.941912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:05.942017 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:05.942493 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:05.942506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:05.942963 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:05.944593 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.944596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.944598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.944846 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.945092 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:05.945095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:05.945097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:05.945191 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:05.945575 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:05.945577 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:05.945680 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:27:05.945792 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 22:27:05.945959 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:05.964726 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:06.078012 19446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:06.078349 19446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:06.078424 19446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:06.083032 19446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:06.083114 19446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:06.083173 19446 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:27:06.083209 19446 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:06.083261 19446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:06.088930 19446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:06.090092 19446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:06.091292 19446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:06.102306 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:06.102280 19451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:06.102272 19452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:06.103012 19446 server_base.cc:1034] running on GCE node
I20260618 22:27:06.103480 19446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:06.104161 19446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:06.105315 19446 hybrid_clock.cc:648] HybridClock initialized: now 1781821626105279 us; error 50 us; skew 500 ppm
Jun 18 22:27:06 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:06.108143 19446 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:06.109061 19446 webserver.cc:460] Webserver started at http://127.16.243.131:35029/ using document root <none> and password file <none>
I20260618 22:27:06.109625 19446 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:06.109712 19446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:06.109920 19446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:06.112119 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "1a38c324a10c4b8bb5aa8588784b184b"
format_stamp: "Formatted at 2026-06-18 22:27:06 on dist-test-slave-jm5g"
server_key: "6f8d9e70139c74d42f87f1cc7bbfad34"
server_key_iv: "e95ee6be145a94039129b12ccec92632"
server_key_version: "encryptionkey@0"
I20260618 22:27:06.112671 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "1a38c324a10c4b8bb5aa8588784b184b"
format_stamp: "Formatted at 2026-06-18 22:27:06 on dist-test-slave-jm5g"
server_key: "6f8d9e70139c74d42f87f1cc7bbfad34"
server_key_iv: "e95ee6be145a94039129b12ccec92632"
server_key_version: "encryptionkey@0"
I20260618 22:27:06.116514 19446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:27:06.119166 19461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:06.120790 19446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:06.120914 19446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "1a38c324a10c4b8bb5aa8588784b184b"
format_stamp: "Formatted at 2026-06-18 22:27:06 on dist-test-slave-jm5g"
server_key: "6f8d9e70139c74d42f87f1cc7bbfad34"
server_key_iv: "e95ee6be145a94039129b12ccec92632"
server_key_version: "encryptionkey@0"
I20260618 22:27:06.121032 19446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:06.154053 19446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:06.157428 19446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:06.157599 19446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:06.159041 19446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:06.159098 19446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:06.159153 19446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:06.159175 19446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:06 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821626, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:06.169826 19446 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:44501
I20260618 22:27:06.169911 19576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:44501 every 8 connection(s)
I20260618 22:27:06.170852 19446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:06.171444 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19446
I20260618 22:27:06.171545 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:06.171856 17358 external_mini_cluster.cc:1428] Setting key 45a7b45a39b65efe05addbe65195871e
I20260618 22:27:06.174048 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.160068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:60273 (local address 127.16.243.190:41939)
0618 22:27:06.160188 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.160192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.161387 (+ 1195us) server_negotiation.cc:407] Connection header received
0618 22:27:06.162676 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.162680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.162742 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.162797 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:06.164430 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.165026 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.166386 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.166673 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.166770 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.170115 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:06.170118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:06.170120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:06.170146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:06.171843 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.172373 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.172377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.172378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.172481 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.172831 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.172835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.172836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.173071 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:06.173160 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.173782 (+ 622us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.173914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:06.175037 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.160666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.131:60273)
0618 22:27:06.161225 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.161271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.162374 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.163158 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.163165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.163674 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:06.164267 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.164282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.165160 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.165163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.166254 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.166264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.166865 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.166869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.167080 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.167844 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:06.167869 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:06.169862 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:06.171966 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.171973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.171990 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.172257 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.172569 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.172572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.172575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.172756 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.173276 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:06.173282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:06.173540 (+ 258us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.173720 (+ 180us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.174071 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":160,"threads_started":1}
I20260618 22:27:06.176153 19577 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41939
I20260618 22:27:06.176302 19577 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:06.176884 19577 heartbeater.cc:511] Master 127.16.243.190:41939 requested a full tablet report, sending...
I20260618 22:27:06.178059 19094 ts_manager.cc:194] Registered new tserver with Master: 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501)
I20260618 22:27:06.178752 19094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:60273
I20260618 22:27:06.185770 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.178509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:54961 (local address 127.16.243.190:41939)
0618 22:27:06.178659 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.178663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.178884 (+ 221us) server_negotiation.cc:407] Connection header received
0618 22:27:06.179011 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.179013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.179050 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.179114 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:06.179650 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.180169 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.181216 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.181547 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.181643 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.182818 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:06.182821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:06.182824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:06.182850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:06.184062 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.184506 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.184509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.184511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.184603 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.184905 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.184909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.184911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.185148 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:06.185249 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.185509 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.185614 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:27:06.185791 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.178412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.16.243.131:54961)
0618 22:27:06.178791 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.178802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.178905 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.179157 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.179161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.179411 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:06.179530 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.179537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.180296 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.180299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.181081 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.181091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.181650 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.181653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.181725 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.182257 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:06.182269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:06.182683 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:06.184178 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.184180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.184182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.184409 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.184688 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.184691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.184693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.184791 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.185244 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:06.185247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:06.185350 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.185441 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.185593 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":17,"thread_start_us":93,"threads_started":1}
I20260618 22:27:06.185928 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:27:06 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:27:06 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:06.210129 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.202413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.16.243.190:41939)
0618 22:27:06.202584 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.202589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.202603 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:06.202645 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.202648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.202687 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.202756 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:06.203486 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.204023 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.204848 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.205068 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.205155 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.207463 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:06.207467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:06.207469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:06.207494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:06.208619 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.209006 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.209009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.209011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.209091 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.209332 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.209336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.209337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.209515 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:06.209630 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.209855 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.209975 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:06.213106 19094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:06.216120 19094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:06.230485 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.224553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42959 (local address 127.0.0.1:44732)
0618 22:27:06.225232 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.225285 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.225416 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.226338 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.226345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.226365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:06.226518 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.226526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.228020 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.228024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.229112 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.229121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.229946 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.229949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.230055 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.230090 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.230199 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.230324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":89,"spinlock_wait_cycles":3456,"thread_start_us":88,"threads_started":1}
I20260618 22:27:06.230486 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.225163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.16.243.130:42959)
0618 22:27:06.225948 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.225959 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.225978 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:27:06.226038 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.226042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.226189 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.226340 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:06.226639 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.227873 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.229239 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.229821 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.229946 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.230102 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.230226 (+ 124us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.230324 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":90,"threads_started":1}
I20260618 22:27:06.230852 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.224345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43475 (local address 127.0.0.1:51960)
0618 22:27:06.225683 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.225695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.225770 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.226623 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.226626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.226642 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:06.226726 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.226732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.228063 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.228066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.229157 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.229168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.230441 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.230445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.230533 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.230579 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.230642 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.230708 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":11,"thread_start_us":104,"threads_started":1}
I20260618 22:27:06.231392 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.224493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.16.243.129:43475)
0618 22:27:06.226063 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.226076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.226098 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:27:06.226165 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.226170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.226332 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.226467 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:06.226838 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.227958 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.229294 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.230148 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.230248 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.230368 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.231006 (+ 638us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.231109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1483,"thread_start_us":90,"threads_started":1}
I20260618 22:27:06.231596 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.224623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44501 (local address 127.0.0.1:34560)
0618 22:27:06.225231 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.225285 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.225421 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.226510 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.226513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.226528 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:06.226637 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.226644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.227706 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.227709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.230026 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.230036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.231216 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.231219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.231316 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.231338 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.231400 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.231475 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":83,"spinlock_wait_cycles":6656,"thread_start_us":87,"threads_started":1}
I20260618 22:27:06.232247 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.225496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.16.243.131:44501)
0618 22:27:06.225786 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.225800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.225825 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:27:06.225936 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.225942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.226190 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.226340 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:06.226767 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.227595 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.230170 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.230994 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.231119 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.231232 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.232016 (+ 784us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.232114 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260618 22:27:06.234041 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 54b50e95423e48508567695e85044221 (DEFAULT_TABLE table=test-table [id=dc02160d78a44c46b9e9940db7b1361c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.234081 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 54b50e95423e48508567695e85044221 (DEFAULT_TABLE table=test-table [id=dc02160d78a44c46b9e9940db7b1361c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.235065 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b50e95423e48508567695e85044221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.235116 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b50e95423e48508567695e85044221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.235209 19511 tablet_service.cc:1467] Processing CreateTablet for tablet 54b50e95423e48508567695e85044221 (DEFAULT_TABLE table=test-table [id=dc02160d78a44c46b9e9940db7b1361c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.236302 19511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b50e95423e48508567695e85044221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.242048 19600 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Bootstrap starting.
I20260618 22:27:06.243129 19601 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Bootstrap starting.
I20260618 22:27:06.243753 19602 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Bootstrap starting.
I20260618 22:27:06.244433 19600 tablet_bootstrap.cc:654] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.245121 19601 tablet_bootstrap.cc:654] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.245303 19600 log.cc:824] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:06.245783 19602 tablet_bootstrap.cc:654] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.245836 19601 log.cc:824] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:06.246484 19602 log.cc:824] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:06.247432 19600 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: No bootstrap required, opened a new log
I20260618 22:27:06.247630 19600 ts_tablet_manager.cc:1397] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:27:06.248234 19602 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: No bootstrap required, opened a new log
I20260618 22:27:06.248396 19602 ts_tablet_manager.cc:1397] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:27:06.248407 19601 tablet_bootstrap.cc:492] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: No bootstrap required, opened a new log
I20260618 22:27:06.248561 19601 ts_tablet_manager.cc:1397] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260618 22:27:06.251250 19602 raft_consensus.cc:348] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.251478 19602 raft_consensus.cc:374] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.251524 19602 raft_consensus.cc:729] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.251514 19600 raft_consensus.cc:348] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.251802 19600 raft_consensus.cc:374] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.251880 19600 raft_consensus.cc:729] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5084f4dcb74ae7a164f39061a8870d, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.252010 19602 consensus_queue.cc:260] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.252295 19600 consensus_queue.cc:260] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.252612 19601 raft_consensus.cc:348] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.252786 19577 heartbeater.cc:503] Master 127.16.243.190:41939 was elected leader, sending a full tablet report...
I20260618 22:27:06.252902 19601 raft_consensus.cc:374] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.253006 19601 raft_consensus.cc:729] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.253235 19602 ts_tablet_manager.cc:1428] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:27:06.253285 19600 ts_tablet_manager.cc:1428] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:27:06.253690 19437 heartbeater.cc:503] Master 127.16.243.190:41939 was elected leader, sending a full tablet report...
I20260618 22:27:06.253811 19601 consensus_queue.cc:260] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.254627 19601 ts_tablet_manager.cc:1428] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:27:06.254941 19297 heartbeater.cc:503] Master 127.16.243.190:41939 was elected leader, sending a full tablet report...
I20260618 22:27:06.413707 19606 raft_consensus.cc:482] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:06.413954 19606 raft_consensus.cc:504] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.415161 19606 leader_election.cc:288] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:06.419103 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.415541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:60973 (local address 127.16.243.130:42959)
0618 22:27:06.415648 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.415652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.415666 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:06.415781 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.415784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.415821 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.415883 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:06.416180 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.416696 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418139 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.418528 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418671 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.418727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.418912 (+ 185us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.418989 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:06.419103 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.415431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42959 (local address 127.16.243.131:60973)
0618 22:27:06.415553 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.415568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.415662 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.415913 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.415916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.415934 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:06.416070 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.416075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.416807 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.416810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.417998 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.418008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.418670 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.418791 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.418810 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.418873 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.418954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"spinlock_wait_cycles":51328}
I20260618 22:27:06.419296 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.415441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43475 (local address 127.16.243.131:43885)
0618 22:27:06.415854 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.415865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.415984 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.416227 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.416230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.416245 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:06.416356 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.416361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.417074 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.417078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.418070 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.418080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.418861 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.418967 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.418989 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.419067 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.419135 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":134,"spinlock_wait_cycles":1792,"thread_start_us":107,"threads_started":1}
I20260618 22:27:06.419271 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.415524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:43885 (local address 127.16.243.129:43475)
0618 22:27:06.415641 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.415646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.415950 (+ 304us) server_negotiation.cc:407] Connection header received
0618 22:27:06.416093 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.416097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.416140 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.416200 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:06.416455 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.416963 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418215 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.418671 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.418864 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.418943 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.419091 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.419167 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:06.419934 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b50e95423e48508567695e85044221" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d" is_pre_election: true
I20260618 22:27:06.419930 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b50e95423e48508567695e85044221" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" is_pre_election: true
I20260618 22:27:06.420213 19385 raft_consensus.cc:2454] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.420214 19251 raft_consensus.cc:2454] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.420733 19463 leader_election.cc:302] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 4c5084f4dcb74ae7a164f39061a8870d; no voters:
I20260618 22:27:06.421016 19606 raft_consensus.cc:2781] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:06.421103 19606 raft_consensus.cc:482] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:06.421142 19606 raft_consensus.cc:3037] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.422278 19606 raft_consensus.cc:504] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.422665 19606 leader_election.cc:288] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Requested vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
W20260618 22:27:06.422909 19578 tablet.cc:2369] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:06.423019 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b50e95423e48508567695e85044221" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d"
I20260618 22:27:06.423136 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b50e95423e48508567695e85044221" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
I20260618 22:27:06.423136 19385 raft_consensus.cc:3037] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.423247 19251 raft_consensus.cc:3037] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.424459 19385 raft_consensus.cc:2454] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.424466 19251 raft_consensus.cc:2454] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.424818 19462 leader_election.cc:302] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:06.425027 19606 raft_consensus.cc:2781] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:06.425254 19606 raft_consensus.cc:686] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 LEADER]: Becoming Leader. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Running, Role: LEADER
I20260618 22:27:06.425592 19606 consensus_queue.cc:237] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.428821 19093 catalog_manager.cc:5495] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131). New cstate: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } health_report { overall_health: HEALTHY } } }
W20260618 22:27:06.432502 19438 tablet.cc:2369] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:06.472213 19298 tablet.cc:2369] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 22:27:06 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821625, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:06.488968 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.480442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41939 (local address 127.0.0.1:37802)
0618 22:27:06.480806 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.480822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.480954 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.481200 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.481203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.481655 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:06.481774 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.481783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.482618 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.482622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.483193 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.483201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.483671 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.483675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.483779 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.484338 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:06.484351 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:06.485828 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:06.487425 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.487436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.487443 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.487675 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.487938 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:06.487941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:06.487943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:06.488019 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:06.488387 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:06.488394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:06.488501 (+ 107us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.488596 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.488765 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260618 22:27:06.488967 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.480488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.16.243.190:41939)
0618 22:27:06.480610 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.480616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.480892 (+ 276us) server_negotiation.cc:407] Connection header received
0618 22:27:06.481049 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.481052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.481096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.481169 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:06.481931 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.482467 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.483325 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.483559 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.483655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.486000 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:06.486003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:06.486005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:06.486026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:06.487291 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.487773 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.487776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.487777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.487847 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:06.488099 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:06.488102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:06.488104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:06.488280 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:06.488360 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.488658 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.488839 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":44}
I20260618 22:27:06.492956 19093 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:37802:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:27:06.493643 19093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:06.498518 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6c36ab7d5041b98628a46fbcd1873e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a57893c8c2845368abd2a287573dd6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:06.498507 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6c36ab7d5041b98628a46fbcd1873e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a57893c8c2845368abd2a287573dd6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:06.499323 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6c36ab7d5041b98628a46fbcd1873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.499327 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6c36ab7d5041b98628a46fbcd1873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.499039 19511 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6c36ab7d5041b98628a46fbcd1873e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a57893c8c2845368abd2a287573dd6f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:06.499607 19511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6c36ab7d5041b98628a46fbcd1873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.502030 19600 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: Bootstrap starting.
I20260618 22:27:06.502305 19602 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: Bootstrap starting.
I20260618 22:27:06.502336 19601 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: Bootstrap starting.
I20260618 22:27:06.502974 19600 tablet_bootstrap.cc:654] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.503279 19602 tablet_bootstrap.cc:654] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.503279 19601 tablet_bootstrap.cc:654] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.504623 19601 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: No bootstrap required, opened a new log
I20260618 22:27:06.504720 19601 ts_tablet_manager.cc:1397] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:06.504954 19602 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: No bootstrap required, opened a new log
I20260618 22:27:06.505038 19602 ts_tablet_manager.cc:1397] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:06.505205 19601 raft_consensus.cc:348] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.505327 19601 raft_consensus.cc:374] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.505368 19601 raft_consensus.cc:729] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.505446 19602 raft_consensus.cc:348] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.505551 19602 raft_consensus.cc:374] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.505510 19601 consensus_queue.cc:260] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.505587 19602 raft_consensus.cc:729] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.505697 19602 consensus_queue.cc:260] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.505784 19601 ts_tablet_manager.cc:1428] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:06.505936 19602 ts_tablet_manager.cc:1428] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:06.505923 19606 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.506093 19606 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.505782 19608 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.506367 19608 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.506597 19600 tablet_bootstrap.cc:492] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: No bootstrap required, opened a new log
I20260618 22:27:06.506692 19600 ts_tablet_manager.cc:1397] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 22:27:06.507130 19600 raft_consensus.cc:348] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.507244 19600 raft_consensus.cc:374] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.507277 19600 raft_consensus.cc:729] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5084f4dcb74ae7a164f39061a8870d, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.507393 19600 consensus_queue.cc:260] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.507630 19600 ts_tablet_manager.cc:1428] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:06.508119 19607 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.508239 19607 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.706499 19606 raft_consensus.cc:482] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:06.706666 19606 raft_consensus.cc:504] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.707206 19606 leader_election.cc:288] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:06.707708 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6c36ab7d5041b98628a46fbcd1873e" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d" is_pre_election: true
I20260618 22:27:06.707849 19385 raft_consensus.cc:2454] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.707798 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6c36ab7d5041b98628a46fbcd1873e" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" is_pre_election: true
I20260618 22:27:06.707953 19251 raft_consensus.cc:2454] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.708209 19463 leader_election.cc:302] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 4c5084f4dcb74ae7a164f39061a8870d; no voters:
I20260618 22:27:06.708374 19606 raft_consensus.cc:2781] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:06.708447 19606 raft_consensus.cc:482] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:06.708494 19606 raft_consensus.cc:3037] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.709452 19606 raft_consensus.cc:504] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.709830 19606 leader_election.cc:288] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Requested vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:06.710240 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6c36ab7d5041b98628a46fbcd1873e" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
I20260618 22:27:06.710235 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6c36ab7d5041b98628a46fbcd1873e" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d"
I20260618 22:27:06.710366 19385 raft_consensus.cc:3037] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.710373 19251 raft_consensus.cc:3037] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.711339 19385 raft_consensus.cc:2454] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.711453 19251 raft_consensus.cc:2454] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.711627 19463 leader_election.cc:302] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 4c5084f4dcb74ae7a164f39061a8870d; no voters:
I20260618 22:27:06.711788 19606 raft_consensus.cc:2781] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:06.711915 19606 raft_consensus.cc:686] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [term 1 LEADER]: Becoming Leader. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Running, Role: LEADER
I20260618 22:27:06.712065 19606 consensus_queue.cc:237] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.712755 19610 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: TxnStatusTablet state changed. Reason: New leader 1a38c324a10c4b8bb5aa8588784b184b. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.712908 19610 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:06.713162 19622 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:27:06.713380 19622 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:27:06.713810 19094 catalog_manager.cc:5495] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131). New cstate: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:06.744765 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.740910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44501 (local address 127.0.0.1:34570)
0618 22:27:06.741151 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.741167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.741279 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.741603 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.741606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.741615 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:06.741712 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.741718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.742470 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.742474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.743142 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.743153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.743640 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.743643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.743727 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.743776 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:06.744464 (+ 688us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:06.744479 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.744542 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.744639 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260618 22:27:06.744801 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.741030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.16.243.131:44501)
0618 22:27:06.741312 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.741317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.741331 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:06.741406 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.741409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.741479 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.741573 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:06.741819 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.742356 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.743266 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.743527 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.743619 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.743923 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:06.744364 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:06.744479 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.744552 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.744608 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260618 22:27:06.750384 19251 raft_consensus.cc:1261] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.750384 19385 raft_consensus.cc:1261] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.751166 19606 consensus_queue.cc:1035] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:06.751655 19606 consensus_queue.cc:1035] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:06.755885 19607 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: TxnStatusTablet state changed. Reason: New leader 1a38c324a10c4b8bb5aa8588784b184b. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.756026 19607 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.755951 19608 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: TxnStatusTablet state changed. Reason: New leader 1a38c324a10c4b8bb5aa8588784b184b. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.756074 19608 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.759879 19610 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.760017 19610 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:06.760942 19608 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.761070 19608 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 855a7a19975e4e17bb92d7d7c3cf3f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.762089 19627 mvcc.cc:205] Tried to move back new op lower bound from 7298341383162208256 to 7298341383013994496. Current Snapshot: MvccSnapshot[applied={T|T < 7298341383162208256}]
I20260618 22:27:06.768746 19606 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.768900 19606 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 1a38c324a10c4b8bb5aa8588784b184b: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:06.769531 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.765533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.16.243.131:44501)
0618 22:27:06.765655 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.765659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.765859 (+ 200us) server_negotiation.cc:407] Connection header received
0618 22:27:06.765981 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.765983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.766031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.766091 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:06.766345 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.767027 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.768380 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.768616 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.768713 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.769078 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:06.769158 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:06.769217 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.769335 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.769386 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:06.770779 19628 mvcc.cc:205] Tried to move back new op lower bound from 7298341383162208256 to 7298341383013994496. Current Snapshot: MvccSnapshot[applied={T|T < 7298341383162208256}]
I20260618 22:27:06.770988 19607 tablet_replica.cc:439] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } } }
I20260618 22:27:06.771102 19607 tablet_replica.cc:442] T 9b6c36ab7d5041b98628a46fbcd1873e P 4c5084f4dcb74ae7a164f39061a8870d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:06.780634 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.777406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:40929 (local address 127.16.243.131:44501)
0618 22:27:06.777532 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:06.777537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:06.777693 (+ 156us) server_negotiation.cc:407] Connection header received
0618 22:27:06.777836 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:06.777839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:06.777880 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:06.777936 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:06.778158 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.778719 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.779550 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.779771 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.779857 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.780195 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:06.780266 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:06.780319 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:06.780403 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 22:27:06.780459 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:06.780951 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:06.777347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44501 (local address 127.16.243.131:40929)
0618 22:27:06.777618 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:06.777629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:06.777731 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:06.777960 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:06.777962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:06.777969 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:06.778071 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.778077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.778811 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.778814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.779441 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:06.779450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:06.779912 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:06.779916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:06.780009 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:06.780051 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:06.780336 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:06.780341 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:06.780398 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 22:27:06.780516 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260618 22:27:06.787757 19385 raft_consensus.cc:1261] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.788106 19251 raft_consensus.cc:1261] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.788170 19635 consensus_queue.cc:1035] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:06.788525 19635 consensus_queue.cc:1035] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:06.793082 19626 mvcc.cc:205] Tried to move back new op lower bound from 7298341383319527424 to 7298341381841596416. Current Snapshot: MvccSnapshot[applied={T|T < 7298341383319527424}]
I20260618 22:27:06.831415 19094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:06.831588 19094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:06.834092 19094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Sending DeleteTablet for 3 replicas of tablet 54b50e95423e48508567695e85044221
I20260618 22:27:06.834921 19231 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b50e95423e48508567695e85044221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:06 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51960
I20260618 22:27:06.834995 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b50e95423e48508567695e85044221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:06 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:44732
I20260618 22:27:06.835062 19510 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b50e95423e48508567695e85044221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:06 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:34560
I20260618 22:27:06.835490 19645 tablet_replica.cc:330] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: stopping tablet replica
I20260618 22:27:06.835825 19645 raft_consensus.cc:2229] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:06.836092 19646 tablet_replica.cc:330] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: stopping tablet replica
I20260618 22:27:06.836299 19645 raft_consensus.cc:2258] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:06.836450 19646 raft_consensus.cc:2229] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:06.836520 19094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:27:06.836645 19646 raft_consensus.cc:2258] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 22:27:06.836963 19094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:06.837014 19644 tablet_replica.cc:330] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: stopping tablet replica
I20260618 22:27:06.837287 19644 raft_consensus.cc:2229] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:06.837530 19644 raft_consensus.cc:2258] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:06.837738 19645 ts_tablet_manager.cc:1905] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:06.838817 19644 ts_tablet_manager.cc:1905] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:06.840785 19645 ts_tablet_manager.cc:1918] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:06.840902 19645 log.cc:1196] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/54b50e95423e48508567695e85044221
I20260618 22:27:06.841212 19645 ts_tablet_manager.cc:1939] T 54b50e95423e48508567695e85044221 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting consensus metadata
I20260618 22:27:06.841778 19644 ts_tablet_manager.cc:1918] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:06.841872 19644 log.cc:1196] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/54b50e95423e48508567695e85044221
I20260618 22:27:06.842159 19644 ts_tablet_manager.cc:1939] T 54b50e95423e48508567695e85044221 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting consensus metadata
I20260618 22:27:06.842448 19081 catalog_manager.cc:4841] TS 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501): tablet 54b50e95423e48508567695e85044221 (table test-table [id=dc02160d78a44c46b9e9940db7b1361c]) successfully deleted
I20260618 22:27:06.843312 19078 catalog_manager.cc:4841] TS 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475): tablet 54b50e95423e48508567695e85044221 (table test-table [id=dc02160d78a44c46b9e9940db7b1361c]) successfully deleted
I20260618 22:27:06.844367 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (DEFAULT_TABLE table=test-table [id=79c5a871bac34ef6b44f38f519585254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.844635 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86aef911209e4b7ba5fa2cc3d260f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.844622 19510 tablet_service.cc:1467] Processing CreateTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (DEFAULT_TABLE table=test-table [id=79c5a871bac34ef6b44f38f519585254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.844797 19646 ts_tablet_manager.cc:1905] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:06.844923 19510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86aef911209e4b7ba5fa2cc3d260f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.847030 19601 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Bootstrap starting.
I20260618 22:27:06.847852 19646 ts_tablet_manager.cc:1918] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:06.848008 19646 log.cc:1196] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/54b50e95423e48508567695e85044221
I20260618 22:27:06.848034 19601 tablet_bootstrap.cc:654] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.848240 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (DEFAULT_TABLE table=test-table [id=79c5a871bac34ef6b44f38f519585254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:06.848489 19646 ts_tablet_manager.cc:1939] T 54b50e95423e48508567695e85044221 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting consensus metadata
I20260618 22:27:06.848523 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86aef911209e4b7ba5fa2cc3d260f2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:06.849009 19602 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Bootstrap starting.
I20260618 22:27:06.849961 19602 tablet_bootstrap.cc:654] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.850440 19600 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Bootstrap starting.
I20260618 22:27:06.851361 19600 tablet_bootstrap.cc:654] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:06.851965 19602 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: No bootstrap required, opened a new log
I20260618 22:27:06.851962 19079 catalog_manager.cc:4841] TS 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959): tablet 54b50e95423e48508567695e85044221 (table test-table [id=dc02160d78a44c46b9e9940db7b1361c]) successfully deleted
I20260618 22:27:06.852052 19602 ts_tablet_manager.cc:1397] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:06.852450 19600 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: No bootstrap required, opened a new log
I20260618 22:27:06.852540 19600 ts_tablet_manager.cc:1397] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:06.852563 19602 raft_consensus.cc:348] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.852696 19602 raft_consensus.cc:374] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.852738 19602 raft_consensus.cc:729] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.852873 19602 consensus_queue.cc:260] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.852968 19600 raft_consensus.cc:348] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.853070 19600 raft_consensus.cc:374] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.853098 19600 raft_consensus.cc:729] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5084f4dcb74ae7a164f39061a8870d, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.853144 19602 ts_tablet_manager.cc:1428] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:06.853209 19600 consensus_queue.cc:260] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.853263 19601 tablet_bootstrap.cc:492] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: No bootstrap required, opened a new log
I20260618 22:27:06.853341 19601 ts_tablet_manager.cc:1397] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:27:06.853447 19600 ts_tablet_manager.cc:1428] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:06.853783 19601 raft_consensus.cc:348] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.853895 19601 raft_consensus.cc:374] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:06.853926 19601 raft_consensus.cc:729] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Initialized, Role: FOLLOWER
I20260618 22:27:06.854038 19601 consensus_queue.cc:260] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.854274 19601 ts_tablet_manager.cc:1428] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:06.911849 19635 raft_consensus.cc:482] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:06.912003 19635 raft_consensus.cc:504] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.912448 19635 leader_election.cc:288] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:06.912909 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86aef911209e4b7ba5fa2cc3d260f2d7" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d" is_pre_election: true
I20260618 22:27:06.912909 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86aef911209e4b7ba5fa2cc3d260f2d7" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" is_pre_election: true
I20260618 22:27:06.913065 19385 raft_consensus.cc:2454] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.913065 19251 raft_consensus.cc:2454] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 0.
I20260618 22:27:06.913391 19463 leader_election.cc:302] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 4c5084f4dcb74ae7a164f39061a8870d, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:06.913602 19635 raft_consensus.cc:2781] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:06.913678 19635 raft_consensus.cc:482] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:06.913715 19635 raft_consensus.cc:3037] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.915016 19635 raft_consensus.cc:504] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.915407 19635 leader_election.cc:288] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Requested vote from peers 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475), 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959)
I20260618 22:27:06.915781 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86aef911209e4b7ba5fa2cc3d260f2d7" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92"
I20260618 22:27:06.915963 19251 raft_consensus.cc:3037] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.916002 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86aef911209e4b7ba5fa2cc3d260f2d7" candidate_uuid: "1a38c324a10c4b8bb5aa8588784b184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d"
I20260618 22:27:06.916121 19385 raft_consensus.cc:3037] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:06.917100 19385 raft_consensus.cc:2454] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.917101 19251 raft_consensus.cc:2454] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a38c324a10c4b8bb5aa8588784b184b in term 1.
I20260618 22:27:06.917416 19463 leader_election.cc:302] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 4c5084f4dcb74ae7a164f39061a8870d; no voters:
I20260618 22:27:06.917604 19635 raft_consensus.cc:2781] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:06.917701 19635 raft_consensus.cc:686] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 LEADER]: Becoming Leader. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Running, Role: LEADER
I20260618 22:27:06.917838 19635 consensus_queue.cc:237] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:06.919795 19092 catalog_manager.cc:5495] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131). New cstate: current_term: 1 leader_uuid: "1a38c324a10c4b8bb5aa8588784b184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:06.948073 19391 raft_consensus.cc:1261] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.948648 19610 consensus_queue.cc:1035] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:06.949090 19251 raft_consensus.cc:1261] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Refusing update from remote peer 1a38c324a10c4b8bb5aa8588784b184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:06.949616 19610 consensus_queue.cc:1035] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:27:06.963658 19626 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:27:08.518404 19094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:08.518596 19094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:08.521435 19094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Sending DeleteTablet for 3 replicas of tablet 86aef911209e4b7ba5fa2cc3d260f2d7
I20260618 22:27:08.522058 19231 tablet_service.cc:1514] Processing DeleteTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51960
I20260618 22:27:08.522186 19510 tablet_service.cc:1514] Processing DeleteTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:34560
I20260618 22:27:08.522428 19673 tablet_replica.cc:330] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: stopping tablet replica
I20260618 22:27:08.522508 19674 tablet_replica.cc:330] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: stopping tablet replica
I20260618 22:27:08.522579 19673 raft_consensus.cc:2229] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:08.522640 19674 raft_consensus.cc:2229] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:08.522701 19673 raft_consensus.cc:2258] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.522826 19674 raft_consensus.cc:2258] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.522929 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 86aef911209e4b7ba5fa2cc3d260f2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:44732
I20260618 22:27:08.523361 19674 ts_tablet_manager.cc:1905] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.523350 19673 ts_tablet_manager.cc:1905] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.526149 19674 ts_tablet_manager.cc:1918] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:27:08.526229 19674 log.cc:1196] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/86aef911209e4b7ba5fa2cc3d260f2d7
I20260618 22:27:08.526505 19674 ts_tablet_manager.cc:1939] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting consensus metadata
I20260618 22:27:08.527063 19673 ts_tablet_manager.cc:1918] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:27:08.527146 19673 log.cc:1196] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/86aef911209e4b7ba5fa2cc3d260f2d7
I20260618 22:27:08.527328 19081 catalog_manager.cc:4841] TS 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501): tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (table test-table [id=79c5a871bac34ef6b44f38f519585254]) successfully deleted
I20260618 22:27:08.527384 19673 ts_tablet_manager.cc:1939] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting consensus metadata
I20260618 22:27:08.528102 19078 catalog_manager.cc:4841] TS 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475): tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (table test-table [id=79c5a871bac34ef6b44f38f519585254]) successfully deleted
I20260618 22:27:08.532029 19675 tablet_replica.cc:330] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: stopping tablet replica
I20260618 22:27:08.532173 19675 raft_consensus.cc:2229] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:08.532310 19675 raft_consensus.cc:2258] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.532928 19675 ts_tablet_manager.cc:1905] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.535657 19675 ts_tablet_manager.cc:1918] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:27:08.535741 19675 log.cc:1196] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/86aef911209e4b7ba5fa2cc3d260f2d7
I20260618 22:27:08.536008 19675 ts_tablet_manager.cc:1939] T 86aef911209e4b7ba5fa2cc3d260f2d7 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting consensus metadata
I20260618 22:27:08.536819 19079 catalog_manager.cc:4841] TS 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959): tablet 86aef911209e4b7ba5fa2cc3d260f2d7 (table test-table [id=79c5a871bac34ef6b44f38f519585254]) successfully deleted
Jun 18 22:27:08 dist-test-slave-jm5g krb5kdc[19049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:27:08 dist-test-slave-jm5g krb5kdc[19049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:08.556432 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.546721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.16.243.190:41939)
0618 22:27:08.546997 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.547004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.547021 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:27:08.547203 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.547207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.547270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.547356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:08.548390 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.549235 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.550307 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.550691 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.550809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.553273 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:08.553276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:08.553278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:08.553305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:08.554874 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:08.555270 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:08.555272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:08.555273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:08.555343 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:08.555681 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:08.555684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:08.555685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:08.555865 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:08.555966 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.556184 (+ 218us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.556286 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260618 22:27:08.558791 19094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:08.559216 19094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:08.563833 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 178e0a626703416f963234dd3bce6356 (DEFAULT_TABLE table=test-table [id=1573dbebabbb4a9c96f1eced195866d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.563871 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 178e0a626703416f963234dd3bce6356 (DEFAULT_TABLE table=test-table [id=1573dbebabbb4a9c96f1eced195866d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.563866 19510 tablet_service.cc:1467] Processing CreateTablet for tablet 178e0a626703416f963234dd3bce6356 (DEFAULT_TABLE table=test-table [id=1573dbebabbb4a9c96f1eced195866d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.564147 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178e0a626703416f963234dd3bce6356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.564147 19510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178e0a626703416f963234dd3bce6356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.564195 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 178e0a626703416f963234dd3bce6356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.566314 19684 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Bootstrap starting.
I20260618 22:27:08.566654 19685 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Bootstrap starting.
I20260618 22:27:08.567550 19685 tablet_bootstrap.cc:654] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.567852 19684 tablet_bootstrap.cc:654] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.568189 19686 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Bootstrap starting.
I20260618 22:27:08.568969 19684 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: No bootstrap required, opened a new log
I20260618 22:27:08.568992 19686 tablet_bootstrap.cc:654] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.569129 19684 ts_tablet_manager.cc:1397] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:08.569226 19685 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: No bootstrap required, opened a new log
I20260618 22:27:08.569314 19685 ts_tablet_manager.cc:1397] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 22:27:08.569603 19684 raft_consensus.cc:348] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.569669 19685 raft_consensus.cc:348] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.569726 19684 raft_consensus.cc:374] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.569761 19684 raft_consensus.cc:729] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5084f4dcb74ae7a164f39061a8870d, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.569761 19685 raft_consensus.cc:374] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.569809 19685 raft_consensus.cc:729] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.569912 19684 consensus_queue.cc:260] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.569940 19685 consensus_queue.cc:260] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.570287 19684 ts_tablet_manager.cc:1428] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.570497 19686 tablet_bootstrap.cc:492] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: No bootstrap required, opened a new log
I20260618 22:27:08.570608 19686 ts_tablet_manager.cc:1397] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:08.570740 19685 ts_tablet_manager.cc:1428] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.570995 19686 raft_consensus.cc:348] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.571076 19686 raft_consensus.cc:374] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.571095 19686 raft_consensus.cc:729] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.571208 19686 consensus_queue.cc:260] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.571565 19686 ts_tablet_manager.cc:1428] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.684871 19691 raft_consensus.cc:482] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:08.685038 19691 raft_consensus.cc:504] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.686147 19691 leader_election.cc:288] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959), 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501)
I20260618 22:27:08.690657 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.686519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:60045 (local address 127.16.243.131:44501)
0618 22:27:08.686787 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.686796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.687008 (+ 212us) server_negotiation.cc:407] Connection header received
0618 22:27:08.687181 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.687185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.687234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.687330 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:08.687600 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.688294 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.689554 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.690040 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.690137 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.690179 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.690404 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.690489 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260618 22:27:08.690652 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.686413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44501 (local address 127.16.243.129:60045)
0618 22:27:08.686901 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:08.686914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:08.687035 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:08.687342 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:08.687344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:08.687357 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:08.687462 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.687468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.688407 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.688410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.689427 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.689437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.690177 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.690181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.690277 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.690303 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:27:08.690371 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:27:08.690462 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260618 22:27:08.690826 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.686368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42959 (local address 127.16.243.129:40587)
0618 22:27:08.686786 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:08.686800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:08.686914 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:08.687186 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:08.687188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:08.687206 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:08.687353 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.687360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.688191 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.688195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.689508 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.689526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.690325 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.690327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.690425 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.690449 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:27:08.690528 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:27:08.690631 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":126,"threads_started":1}
I20260618 22:27:08.690995 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.686476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:40587 (local address 127.16.243.130:42959)
0618 22:27:08.686797 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.686806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.686886 (+ 80us) server_negotiation.cc:407] Connection header received
0618 22:27:08.687009 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.687012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.687062 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.687143 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:08.687461 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.688068 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.689757 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.690202 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.690322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.690371 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.690594 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.690675 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":138,"threads_started":1}
I20260618 22:27:08.691242 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178e0a626703416f963234dd3bce6356" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a38c324a10c4b8bb5aa8588784b184b" is_pre_election: true
I20260618 22:27:08.691326 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178e0a626703416f963234dd3bce6356" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d" is_pre_election: true
I20260618 22:27:08.691444 19391 raft_consensus.cc:2454] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 0.
I20260618 22:27:08.691447 19531 raft_consensus.cc:2454] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 0.
I20260618 22:27:08.691854 19185 leader_election.cc:302] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:08.692128 19691 raft_consensus.cc:2781] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:08.692184 19691 raft_consensus.cc:482] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:08.692222 19691 raft_consensus.cc:3037] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.693241 19691 raft_consensus.cc:504] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.693616 19691 leader_election.cc:288] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959), 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501)
I20260618 22:27:08.693993 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178e0a626703416f963234dd3bce6356" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a38c324a10c4b8bb5aa8588784b184b"
I20260618 22:27:08.694013 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "178e0a626703416f963234dd3bce6356" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d"
I20260618 22:27:08.694111 19531 raft_consensus.cc:3037] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.694120 19391 raft_consensus.cc:3037] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.695111 19391 raft_consensus.cc:2454] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 1.
I20260618 22:27:08.695185 19531 raft_consensus.cc:2454] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 1.
I20260618 22:27:08.695427 19183 leader_election.cc:302] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5084f4dcb74ae7a164f39061a8870d, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:08.695590 19691 raft_consensus.cc:2781] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:08.695811 19691 raft_consensus.cc:686] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 LEADER]: Becoming Leader. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Running, Role: LEADER
I20260618 22:27:08.696082 19691 consensus_queue.cc:237] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.698308 19094 catalog_manager.cc:5495] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:08.732772 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.728455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.16.243.129:43475)
0618 22:27:08.728732 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.728740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.728753 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:08.728834 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.728838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.728900 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.729031 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:08.729274 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.730130 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.731043 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.731419 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.731514 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.731734 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:08.732308 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:08.732415 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.732534 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.732594 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":128,"threads_started":1}
I20260618 22:27:08.743077 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.740114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:54743 (local address 127.16.243.131:44501)
0618 22:27:08.740239 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.740243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.740381 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:27:08.740502 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.740505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.740549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.740614 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:08.740821 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.741321 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.742109 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.742315 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.742382 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.742681 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:08.742761 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:08.742823 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.742899 (+ 76us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.742953 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:08.743098 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.740006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44501 (local address 127.16.243.129:54743)
0618 22:27:08.740308 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:08.740320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:08.740414 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:08.740633 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:08.740635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:08.740641 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:08.740749 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.740753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.741400 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.741402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.742022 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.742030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.742400 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.742403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.742477 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.742522 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:08.742838 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:08.742843 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:08.742891 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 22:27:08.742959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
I20260618 22:27:08.749181 19391 raft_consensus.cc:1261] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Refusing update from remote peer 855a7a19975e4e17bb92d7d7c3cf3f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:08.749354 19531 raft_consensus.cc:1261] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Refusing update from remote peer 855a7a19975e4e17bb92d7d7c3cf3f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:08.749694 19696 consensus_queue.cc:1035] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:08.750243 19696 consensus_queue.cc:1035] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:08.767241 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.763885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:46851 (local address 127.16.243.129:43475)
0618 22:27:08.764151 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:08.764156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:08.764229 (+ 73us) server_negotiation.cc:407] Connection header received
0618 22:27:08.764381 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:08.764384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:08.764434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:08.764495 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:08.764726 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.765241 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.766087 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.766371 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.766476 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.766764 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:08.766864 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:08.766926 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:08.767082 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:27:08.767129 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260618 22:27:08.767253 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:08.763847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43475 (local address 127.16.243.131:46851)
0618 22:27:08.764150 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:08.764165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:08.764272 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:08.764514 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:08.764517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:08.764525 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:08.764627 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.764632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.765353 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.765357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.765966 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:08.765975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:08.766487 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:08.766490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:08.766612 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:08.766646 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:08.766994 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:08.767001 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:27:08.767058 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 22:27:08.767132 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260618 22:27:08.790768 19092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:08.790923 19092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:08.793246 19092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Sending DeleteTablet for 3 replicas of tablet 178e0a626703416f963234dd3bce6356
I20260618 22:27:08.793922 19231 tablet_service.cc:1514] Processing DeleteTablet for tablet 178e0a626703416f963234dd3bce6356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51960
I20260618 22:27:08.793929 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 178e0a626703416f963234dd3bce6356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:44732
I20260618 22:27:08.794116 19673 tablet_replica.cc:330] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: stopping tablet replica
I20260618 22:27:08.794144 19675 tablet_replica.cc:330] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: stopping tablet replica
I20260618 22:27:08.794227 19673 raft_consensus.cc:2229] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:08.794274 19675 raft_consensus.cc:2229] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:08.794405 19675 raft_consensus.cc:2258] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.794580 19673 raft_consensus.cc:2258] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.795239 19675 ts_tablet_manager.cc:1905] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.795215 19510 tablet_service.cc:1514] Processing DeleteTablet for tablet 178e0a626703416f963234dd3bce6356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:08 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:34560
I20260618 22:27:08.795369 19673 ts_tablet_manager.cc:1905] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.795420 19674 tablet_replica.cc:330] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: stopping tablet replica
I20260618 22:27:08.795554 19674 raft_consensus.cc:2229] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:08.795708 19674 raft_consensus.cc:2258] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:08.796648 19674 ts_tablet_manager.cc:1905] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:08.797747 19673 ts_tablet_manager.cc:1918] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:08.797747 19675 ts_tablet_manager.cc:1918] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:08.797813 19673 log.cc:1196] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/178e0a626703416f963234dd3bce6356
I20260618 22:27:08.797813 19675 log.cc:1196] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/178e0a626703416f963234dd3bce6356
I20260618 22:27:08.798050 19673 ts_tablet_manager.cc:1939] T 178e0a626703416f963234dd3bce6356 P 855a7a19975e4e17bb92d7d7c3cf3f92: Deleting consensus metadata
I20260618 22:27:08.798050 19675 ts_tablet_manager.cc:1939] T 178e0a626703416f963234dd3bce6356 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting consensus metadata
I20260618 22:27:08.798805 19079 catalog_manager.cc:4841] TS 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959): tablet 178e0a626703416f963234dd3bce6356 (table test-table [id=1573dbebabbb4a9c96f1eced195866d9]) successfully deleted
I20260618 22:27:08.799068 19092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:08.799441 19092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:08.799522 19674 ts_tablet_manager.cc:1918] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:08.799597 19674 log.cc:1196] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/178e0a626703416f963234dd3bce6356
I20260618 22:27:08.799863 19674 ts_tablet_manager.cc:1939] T 178e0a626703416f963234dd3bce6356 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting consensus metadata
I20260618 22:27:08.800570 19078 catalog_manager.cc:4841] TS 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475): tablet 178e0a626703416f963234dd3bce6356 (table test-table [id=1573dbebabbb4a9c96f1eced195866d9]) successfully deleted
I20260618 22:27:08.801266 19081 catalog_manager.cc:4841] TS 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501): tablet 178e0a626703416f963234dd3bce6356 (table test-table [id=1573dbebabbb4a9c96f1eced195866d9]) successfully deleted
I20260618 22:27:08.803930 19231 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 (DEFAULT_TABLE table=test-table [id=542c23685f0b4a80a09f81d66c772229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.804205 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.804163 19510 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 (DEFAULT_TABLE table=test-table [id=542c23685f0b4a80a09f81d66c772229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.804376 19510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.806699 19685 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Bootstrap starting.
I20260618 22:27:08.807027 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 (DEFAULT_TABLE table=test-table [id=542c23685f0b4a80a09f81d66c772229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:08.807286 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:08.807627 19685 tablet_bootstrap.cc:654] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.808179 19686 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92: Bootstrap starting.
I20260618 22:27:08.809167 19686 tablet_bootstrap.cc:654] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.809170 19684 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Bootstrap starting.
I20260618 22:27:08.809937 19684 tablet_bootstrap.cc:654] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:08.810300 19685 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: No bootstrap required, opened a new log
I20260618 22:27:08.810398 19685 ts_tablet_manager.cc:1397] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:27:08.810868 19685 raft_consensus.cc:348] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.810992 19686 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92: No bootstrap required, opened a new log
I20260618 22:27:08.810997 19685 raft_consensus.cc:374] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.811033 19685 raft_consensus.cc:729] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a38c324a10c4b8bb5aa8588784b184b, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.811072 19686 ts_tablet_manager.cc:1397] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 22:27:08.811168 19685 consensus_queue.cc:260] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.811456 19685 ts_tablet_manager.cc:1428] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.811506 19686 raft_consensus.cc:348] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.811615 19686 raft_consensus.cc:374] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.811643 19686 raft_consensus.cc:729] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.811754 19686 consensus_queue.cc:260] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.811993 19686 ts_tablet_manager.cc:1428] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.812911 19684 tablet_bootstrap.cc:492] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: No bootstrap required, opened a new log
I20260618 22:27:08.813000 19684 ts_tablet_manager.cc:1397] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 22:27:08.813416 19684 raft_consensus.cc:348] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.813549 19684 raft_consensus.cc:374] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:08.813585 19684 raft_consensus.cc:729] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5084f4dcb74ae7a164f39061a8870d, State: Initialized, Role: FOLLOWER
I20260618 22:27:08.813709 19684 consensus_queue.cc:260] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.813957 19684 ts_tablet_manager.cc:1428] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:08.835125 19705 raft_consensus.cc:482] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:08.835273 19705 raft_consensus.cc:504] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.835698 19705 leader_election.cc:288] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959), 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501)
I20260618 22:27:08.836153 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2ed4be18a54a28b6b1e0d83a62a0c1" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d" is_pre_election: true
I20260618 22:27:08.836154 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2ed4be18a54a28b6b1e0d83a62a0c1" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a38c324a10c4b8bb5aa8588784b184b" is_pre_election: true
I20260618 22:27:08.836306 19391 raft_consensus.cc:2454] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 0.
I20260618 22:27:08.836321 19531 raft_consensus.cc:2454] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 0.
I20260618 22:27:08.836627 19185 leader_election.cc:302] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:08.836831 19705 raft_consensus.cc:2781] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:08.836892 19705 raft_consensus.cc:482] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:08.836920 19705 raft_consensus.cc:3037] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.838335 19705 raft_consensus.cc:504] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.838723 19705 leader_election.cc:288] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959), 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501)
I20260618 22:27:08.839126 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2ed4be18a54a28b6b1e0d83a62a0c1" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5084f4dcb74ae7a164f39061a8870d"
I20260618 22:27:08.839129 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2ed4be18a54a28b6b1e0d83a62a0c1" candidate_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a38c324a10c4b8bb5aa8588784b184b"
I20260618 22:27:08.839246 19391 raft_consensus.cc:3037] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.839280 19531 raft_consensus.cc:3037] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:08.840301 19391 raft_consensus.cc:2454] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 1.
I20260618 22:27:08.840468 19531 raft_consensus.cc:2454] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855a7a19975e4e17bb92d7d7c3cf3f92 in term 1.
I20260618 22:27:08.840801 19185 leader_election.cc:302] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a38c324a10c4b8bb5aa8588784b184b, 855a7a19975e4e17bb92d7d7c3cf3f92; no voters:
I20260618 22:27:08.841176 19705 raft_consensus.cc:2781] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:08.841271 19705 raft_consensus.cc:686] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [term 1 LEADER]: Becoming Leader. State: Replica: 855a7a19975e4e17bb92d7d7c3cf3f92, State: Running, Role: LEADER
I20260618 22:27:08.841398 19705 consensus_queue.cc:237] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } }
I20260618 22:27:08.843082 19094 catalog_manager.cc:5495] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855a7a19975e4e17bb92d7d7c3cf3f92" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:08.864907 19531 raft_consensus.cc:1261] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Refusing update from remote peer 855a7a19975e4e17bb92d7d7c3cf3f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:08.865056 19391 raft_consensus.cc:1261] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Refusing update from remote peer 855a7a19975e4e17bb92d7d7c3cf3f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:08.865294 19705 consensus_queue.cc:1035] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a38c324a10c4b8bb5aa8588784b184b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:08.865480 19696 consensus_queue.cc:1035] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 855a7a19975e4e17bb92d7d7c3cf3f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5084f4dcb74ae7a164f39061a8870d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:27:08.879107 19657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:27:10.433449 19094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:10.433615 19094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:10.437198 19094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d1cfea21ff4422a90803d113805db3: Sending DeleteTablet for 3 replicas of tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1
I20260618 22:27:10.437882 19510 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:10 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:34560
I20260618 22:27:10.437881 19231 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:10 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51960
I20260618 22:27:10.438257 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:10 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:44732
I20260618 22:27:10.438372 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19166
I20260618 22:27:10.438257 19737 tablet_replica.cc:330] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: stopping tablet replica
I20260618 22:27:10.439584 19739 tablet_replica.cc:330] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: stopping tablet replica
I20260618 22:27:10.439715 19739 raft_consensus.cc:2229] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:10.439725 19737 raft_consensus.cc:2229] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:10.439849 19739 raft_consensus.cc:2258] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:10.439862 19737 raft_consensus.cc:2258] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:10.440451 19739 ts_tablet_manager.cc:1905] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:10.442685 19739 ts_tablet_manager.cc:1918] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:27:10.442760 19739 log.cc:1196] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/0a2ed4be18a54a28b6b1e0d83a62a0c1
I20260618 22:27:10.442983 19739 ts_tablet_manager.cc:1939] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 4c5084f4dcb74ae7a164f39061a8870d: Deleting consensus metadata
I20260618 22:27:10.443756 19079 catalog_manager.cc:4841] TS 4c5084f4dcb74ae7a164f39061a8870d (127.16.243.130:42959): tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 (table test-table [id=542c23685f0b4a80a09f81d66c772229]) successfully deleted
I20260618 22:27:10.444021 19737 ts_tablet_manager.cc:1905] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:27:10.444686 19078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:43475 (error 108)
I20260618 22:27:10.444907 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19306
I20260618 22:27:10.447839 19737 ts_tablet_manager.cc:1918] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:27:10.447926 19737 log.cc:1196] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/0a2ed4be18a54a28b6b1e0d83a62a0c1
I20260618 22:27:10.448132 19737 ts_tablet_manager.cc:1939] T 0a2ed4be18a54a28b6b1e0d83a62a0c1 P 1a38c324a10c4b8bb5aa8588784b184b: Deleting consensus metadata
I20260618 22:27:10.448870 19081 catalog_manager.cc:4841] TS 1a38c324a10c4b8bb5aa8588784b184b (127.16.243.131:44501): tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1 (table test-table [id=542c23685f0b4a80a09f81d66c772229]) successfully deleted
I20260618 22:27:10.450268 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:10.445839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43475 (local address 127.0.0.1:59546)
0618 22:27:10.450102 (+ 4263us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:10.450177 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:43475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4185,"thread_start_us":81,"threads_started":1}
W20260618 22:27:10.450407 19078 catalog_manager.cc:4584] TS 855a7a19975e4e17bb92d7d7c3cf3f92 (127.16.243.129:43475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a2ed4be18a54a28b6b1e0d83a62a0c1: Network error: Client connection negotiation failed: client connection to 127.16.243.129:43475: connect: Connection refused (error 111)
I20260618 22:27:10.450529 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19446
I20260618 22:27:10.457152 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19062
2026-06-18T22:27:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6276 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:10 dist-test-slave-jm5g krb5kdc[19744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:10 dist-test-slave-jm5g krb5kdc[19744](info): set up 2 sockets
Jun 18 22:27:10 dist-test-slave-jm5g krb5kdc[19744](info): commencing operation
krb5kdc: starting...
W20260618 22:27:11.418193 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:11 dist-test-slave-jm5g krb5kdc[19744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:11.536882 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34121
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:34121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:11.649928 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:11.650252 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:11.650327 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:11.654414 19757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:11.654508 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:11.654543 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:11.654593 19757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:11.654620 19757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:11.660175 19757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:34121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:11.661327 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:11.662416 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:11.671334 19763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:11.671422 19757 server_base.cc:1034] running on GCE node
W20260618 22:27:11.671331 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:11.671331 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:11.672202 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:11.673399 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:11.674593 19757 hybrid_clock.cc:648] HybridClock initialized: now 1781821631674572 us; error 33 us; skew 500 ppm
Jun 18 22:27:11 dist-test-slave-jm5g krb5kdc[19744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821631, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:11.677366 19757 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:11.678287 19757 webserver.cc:460] Webserver started at http://127.16.243.190:37893/ using document root <none> and password file <none>
I20260618 22:27:11.678898 19757 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:11.678963 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:11.679145 19757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:11.681265 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "beb4aee17f7541d69db3dfd922b6cb36"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "e38844ac20b053a5bbbaab6be1151c5f"
server_key_iv: "36dc3a78bc0e5bb38823aec93ec2f45f"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.681838 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "beb4aee17f7541d69db3dfd922b6cb36"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "e38844ac20b053a5bbbaab6be1151c5f"
server_key_iv: "36dc3a78bc0e5bb38823aec93ec2f45f"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.686141 19757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260618 22:27:11.688877 19772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:11.690303 19757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:11.690433 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "beb4aee17f7541d69db3dfd922b6cb36"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "e38844ac20b053a5bbbaab6be1151c5f"
server_key_iv: "36dc3a78bc0e5bb38823aec93ec2f45f"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.690543 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:11.708801 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:11.711962 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:11.712136 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:11.719025 19757 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:34121
I20260618 22:27:11.719058 19824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:34121 every 8 connection(s)
I20260618 22:27:11.720072 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:11.722908 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:11.723143 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19757
I20260618 22:27:11.723245 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:11.723552 17358 external_mini_cluster.cc:1428] Setting key c9a26e860a9a798f91908141cb3f3675
I20260618 22:27:11.730244 19825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36: Bootstrap starting.
Jun 18 22:27:11 dist-test-slave-jm5g krb5kdc[19744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:11.733074 19825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:11.733947 19825 log.cc:824] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:11.736143 19825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36: No bootstrap required, opened a new log
I20260618 22:27:11.738248 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:11.725047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.16.243.190:34121)
0618 22:27:11.725509 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:11.725527 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:11.725744 (+ 217us) server_negotiation.cc:407] Connection header received
0618 22:27:11.726513 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:11.726532 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:11.726921 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:11.727215 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:11.728094 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.729080 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.730228 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.730811 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.730931 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:11.733576 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:11.733583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:11.733601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:11.733635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:11.735432 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:11.735913 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:11.735917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:11.735922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:11.736028 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:11.736308 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:11.736311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:11.736313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:11.736730 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:11.736878 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:11.737192 (+ 314us) server_negotiation.cc:299] Negotiation successful
0618 22:27:11.737468 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260618 22:27:11.740127 19825 raft_consensus.cc:348] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } }
I20260618 22:27:11.740408 19825 raft_consensus.cc:374] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:11.740468 19825 raft_consensus.cc:729] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb4aee17f7541d69db3dfd922b6cb36, State: Initialized, Role: FOLLOWER
I20260618 22:27:11.740904 19825 consensus_queue.cc:260] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } }
I20260618 22:27:11.741031 19825 raft_consensus.cc:388] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:11.741092 19825 raft_consensus.cc:482] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:11.741180 19825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:11.742125 19825 raft_consensus.cc:504] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } }
I20260618 22:27:11.742475 19825 leader_election.cc:302] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beb4aee17f7541d69db3dfd922b6cb36; no voters:
I20260618 22:27:11.742831 19825 leader_election.cc:288] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:11.742952 19830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:11.743162 19830 raft_consensus.cc:686] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [term 1 LEADER]: Becoming Leader. State: Replica: beb4aee17f7541d69db3dfd922b6cb36, State: Running, Role: LEADER
I20260618 22:27:11.743481 19830 consensus_queue.cc:237] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } }
I20260618 22:27:11.744017 19825 sys_catalog.cc:564] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:11.745040 19832 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beb4aee17f7541d69db3dfd922b6cb36. Latest consensus state: current_term: 1 leader_uuid: "beb4aee17f7541d69db3dfd922b6cb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } } }
I20260618 22:27:11.745164 19832 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:11.745491 19831 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beb4aee17f7541d69db3dfd922b6cb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb4aee17f7541d69db3dfd922b6cb36" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34121 } } }
I20260618 22:27:11.745620 19831 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:11.745544 19839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:11.748847 19839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:11.754528 19839 catalog_manager.cc:1347] Generated new cluster ID: 6829a1f944c347a7982a4f207cb7dc17
I20260618 22:27:11.754649 19839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:11.764760 19839 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:11.765933 19839 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:11.772779 19839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beb4aee17f7541d69db3dfd922b6cb36: Generated new TSK 0
I20260618 22:27:11.773480 19839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:11.801748 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:11.913560 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:11.913873 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:11.913934 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:11.918093 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:11.918179 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:11.918272 19851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:11.923888 19851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:11.925015 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:11.926041 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:11.935688 19856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:11.935688 19859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:11.935729 19851 server_base.cc:1034] running on GCE node
W20260618 22:27:11.935688 19857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:11.936414 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:11.937101 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:11.938261 19851 hybrid_clock.cc:648] HybridClock initialized: now 1781821631938237 us; error 39 us; skew 500 ppm
Jun 18 22:27:11 dist-test-slave-jm5g krb5kdc[19744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821631, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:11.941054 19851 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:11.941960 19851 webserver.cc:460] Webserver started at http://127.16.243.129:41217/ using document root <none> and password file <none>
I20260618 22:27:11.942508 19851 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:11.942621 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:11.942836 19851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:11.945007 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "1fc6defe970e4fbfa1c2d0d490be45f3"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "7d3dd56a4cec72f7d88fc9936b90a7d7"
server_key_iv: "c090a6b7bc52afb3687ee92026542bed"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.945603 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "1fc6defe970e4fbfa1c2d0d490be45f3"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "7d3dd56a4cec72f7d88fc9936b90a7d7"
server_key_iv: "c090a6b7bc52afb3687ee92026542bed"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.949393 19851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260618 22:27:11.951932 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:11.953161 19851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:11.953274 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "1fc6defe970e4fbfa1c2d0d490be45f3"
format_stamp: "Formatted at 2026-06-18 22:27:11 on dist-test-slave-jm5g"
server_key: "7d3dd56a4cec72f7d88fc9936b90a7d7"
server_key_iv: "c090a6b7bc52afb3687ee92026542bed"
server_key_version: "encryptionkey@0"
I20260618 22:27:11.953372 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:11.964303 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:11.967617 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:11.967801 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:11.968438 19851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:11.969249 19851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:11.969293 19851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:11.969331 19851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:11.969345 19851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:11.977766 19851 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:39461
I20260618 22:27:11.977788 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:39461 every 8 connection(s)
I20260618 22:27:11.978760 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:27:11 dist-test-slave-jm5g krb5kdc[19744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821631, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:11.988338 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19851
I20260618 22:27:11.988459 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:11.988752 17358 external_mini_cluster.cc:1428] Setting key 5717ff4066c658ddf2a5e3b941ba8dfd
I20260618 22:27:11.992429 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:11.980545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:33959 (local address 127.16.243.190:34121)
0618 22:27:11.980684 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:11.980688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:11.981547 (+ 859us) server_negotiation.cc:407] Connection header received
0618 22:27:11.982791 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:11.982794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:11.982843 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:11.982915 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:11.984082 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.984612 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.985726 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.985963 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.986050 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:11.988752 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:11.988756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:11.988758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:11.988785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:11.990201 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:11.990831 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:11.990833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:11.990834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:11.990909 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:11.991252 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:11.991254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:11.991255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:11.991423 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:11.991492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:11.992119 (+ 627us) server_negotiation.cc:299] Negotiation successful
0618 22:27:11.992261 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:27:11.993295 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:11.980945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34121 (local address 127.16.243.129:33959)
0618 22:27:11.981393 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:11.981437 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:11.982601 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:11.983086 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:11.983092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:11.983445 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:11.983946 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:11.983960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.984704 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.984706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:11.985616 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:11.985625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:11.986096 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:11.986098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:11.986248 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:11.986804 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:11.986821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:11.988563 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:11.990317 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:11.990324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:11.990344 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:11.990750 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:11.990983 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:11.990986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:11.990988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:11.991174 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:11.991511 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:11.991518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:11.991869 (+ 351us) client_negotiation.cc:769] Sending connection context
0618 22:27:11.992037 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:27:11.992315 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260618 22:27:11.994570 19980 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34121
I20260618 22:27:11.994808 19980 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:11.995343 19980 heartbeater.cc:511] Master 127.16.243.190:34121 requested a full tablet report, sending...
I20260618 22:27:11.996812 19783 ts_manager.cc:194] Registered new tserver with Master: 1fc6defe970e4fbfa1c2d0d490be45f3 (127.16.243.129:39461)
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:11.997956 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:33959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:12.013867 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:12.124987 19985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:12.125274 19985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:12.125319 19985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:12.129520 19985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:12.129590 19985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:12.129668 19985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:12.135596 19985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:12.136729 19985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:12.137768 19985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:12.147303 19991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:12.147312 19990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:12.147312 19993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:12.147559 19985 server_base.cc:1034] running on GCE node
I20260618 22:27:12.148149 19985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:12.148828 19985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:12.150022 19985 hybrid_clock.cc:648] HybridClock initialized: now 1781821632149998 us; error 40 us; skew 500 ppm
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[19744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821632, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:12.152791 19985 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:12.153738 19985 webserver.cc:460] Webserver started at http://127.16.243.130:35997/ using document root <none> and password file <none>
I20260618 22:27:12.154309 19985 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:12.154376 19985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:12.154614 19985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:12.156747 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "19f821ece58c41f3bd157a7160165c44"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "658cf66f296a7384862915a05d526187"
server_key_iv: "5c21ca26613689f360ffcb4525cd7b10"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.157341 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "19f821ece58c41f3bd157a7160165c44"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "658cf66f296a7384862915a05d526187"
server_key_iv: "5c21ca26613689f360ffcb4525cd7b10"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.161113 19985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:27:12.163621 20000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.164888 19985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:12.165019 19985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "19f821ece58c41f3bd157a7160165c44"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "658cf66f296a7384862915a05d526187"
server_key_iv: "5c21ca26613689f360ffcb4525cd7b10"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.165127 19985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:12.189885 19985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:12.193164 19985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:12.193364 19985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:12.193955 19985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:12.194861 19985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:12.194914 19985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.194955 19985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:12.194975 19985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.203275 19985 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:38599
I20260618 22:27:12.203301 20113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:38599 every 8 connection(s)
I20260618 22:27:12.204188 19985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:12.209784 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 19985
I20260618 22:27:12.209903 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:12.210175 17358 external_mini_cluster.cc:1428] Setting key 4fa6dc45034059aeac033f8a77784bad
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[19744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821632, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:12.217695 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:12.205948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55859 (local address 127.16.243.190:34121)
0618 22:27:12.206077 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:12.206081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:12.207144 (+ 1063us) server_negotiation.cc:407] Connection header received
0618 22:27:12.208016 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:12.208020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:12.208075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:12.208146 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:12.209306 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.209974 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.211255 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.211483 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.211593 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:12.214074 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:12.214077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:12.214079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:12.214105 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:12.215476 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:12.216091 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:12.216093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:12.216094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:12.216163 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:12.216525 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:12.216527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:12.216529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:12.216695 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:12.216761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:12.217447 (+ 686us) server_negotiation.cc:299] Negotiation successful
0618 22:27:12.217561 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:12.218892 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:12.206337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34121 (local address 127.16.243.130:55859)
0618 22:27:12.206987 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:12.207029 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:12.207814 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:12.208318 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:12.208325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:12.208688 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:12.209170 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:12.209183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.210096 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.210099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:12.211149 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:12.211159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.211633 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.211636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:12.211780 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:12.212344 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:12.212361 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:12.213889 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:12.215600 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:12.215607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:12.215626 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:12.215980 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:12.216240 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:12.216243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:12.216245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:12.216440 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:12.216790 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:12.216797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:12.217141 (+ 344us) client_negotiation.cc:769] Sending connection context
0618 22:27:12.217368 (+ 227us) client_negotiation.cc:240] Negotiation successful
0618 22:27:12.217738 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":137,"threads_started":1}
I20260618 22:27:12.220041 20114 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34121
I20260618 22:27:12.220273 20114 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:12.220721 20114 heartbeater.cc:511] Master 127.16.243.190:34121 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:12.221735 19783 ts_manager.cc:194] Registered new tserver with Master: 19f821ece58c41f3bd157a7160165c44 (127.16.243.130:38599)
I20260618 22:27:12.222297 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:55859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:12.236931 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:12.346454 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:12.346757 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:12.346819 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:12.350827 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:12.350893 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:12.350970 20119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:12.356235 20119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:12.357323 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:12.358330 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:12.367772 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:12.367868 20119 server_base.cc:1034] running on GCE node
W20260618 22:27:12.368005 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:12.367774 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:12.368497 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:12.369169 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:12.370354 20119 hybrid_clock.cc:648] HybridClock initialized: now 1781821632370338 us; error 25 us; skew 500 ppm
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[19744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821632, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:12.373190 20119 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:12.374114 20119 webserver.cc:460] Webserver started at http://127.16.243.131:46299/ using document root <none> and password file <none>
I20260618 22:27:12.374850 20119 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:12.374936 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:12.375159 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:12.377334 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "12fb6b4edb654f4183a2d4b655423f60"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "1aee43e9c18207c4b1c31d48592d24a3"
server_key_iv: "2b2a6102fe3dc82fe24ee0158e50f6c5"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.377961 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "12fb6b4edb654f4183a2d4b655423f60"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "1aee43e9c18207c4b1c31d48592d24a3"
server_key_iv: "2b2a6102fe3dc82fe24ee0158e50f6c5"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.382073 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:12.385007 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.386435 20119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:12.386588 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "12fb6b4edb654f4183a2d4b655423f60"
format_stamp: "Formatted at 2026-06-18 22:27:12 on dist-test-slave-jm5g"
server_key: "1aee43e9c18207c4b1c31d48592d24a3"
server_key_iv: "2b2a6102fe3dc82fe24ee0158e50f6c5"
server_key_version: "encryptionkey@0"
I20260618 22:27:12.386706 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:12.406391 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:12.409806 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:12.410014 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:12.410647 20119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:12.411505 20119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:12.411562 20119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.411614 20119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:12.411643 20119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:12.420083 20119 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:34107
I20260618 22:27:12.420110 20247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:34107 every 8 connection(s)
I20260618 22:27:12.421010 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:12.423231 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20119
I20260618 22:27:12.423341 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:12.423614 17358 external_mini_cluster.cc:1428] Setting key 30c469c3eba82dee9be9376273070e89
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[19744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821632, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:12.434675 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:12.422937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:46159 (local address 127.16.243.190:34121)
0618 22:27:12.423113 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:12.423118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:12.424295 (+ 1177us) server_negotiation.cc:407] Connection header received
0618 22:27:12.425382 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:12.425386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:12.425444 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:12.425540 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:12.426898 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.427428 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.428595 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.428826 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.428918 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:12.431395 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:12.431398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:12.431400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:12.431429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:12.432757 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:12.433259 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:12.433262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:12.433264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:12.433344 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:12.433632 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:12.433634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:12.433635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:12.433823 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:12.433896 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:12.434408 (+ 512us) server_negotiation.cc:299] Negotiation successful
0618 22:27:12.434515 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:27:12.435586 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:12.423331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34121 (local address 127.16.243.131:46159)
0618 22:27:12.424120 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:12.424178 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:12.425153 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:12.425697 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:12.425707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:12.426165 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:12.426717 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:12.426732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.427537 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.427540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:12.428462 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:12.428472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:12.428970 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:12.428973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:12.429118 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:12.429654 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:12.429672 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:12.431202 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:12.432869 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:12.432875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:12.432888 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:12.433152 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:12.433418 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:12.433420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:12.433421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:12.433550 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:12.433942 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:12.433948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:12.434214 (+ 266us) client_negotiation.cc:769] Sending connection context
0618 22:27:12.434375 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:27:12.434700 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":142,"threads_started":1}
I20260618 22:27:12.436803 20248 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34121
I20260618 22:27:12.437048 20248 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:12.437538 20248 heartbeater.cc:511] Master 127.16.243.190:34121 requested a full tablet report, sending...
I20260618 22:27:12.438591 19783 ts_manager.cc:194] Registered new tserver with Master: 12fb6b4edb654f4183a2d4b655423f60 (127.16.243.131:34107)
I20260618 22:27:12.439232 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:46159
I20260618 22:27:12.449313 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:27:12.461512 19828 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:12.460156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.16.243.190:34121)
0618 22:27:12.460311 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:12.460317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:12.460435 (+ 118us) server_negotiation.cc:407] Connection header received
0618 22:27:12.460549 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:12.460552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:12.460596 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:12.460664 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:12.461358 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43844: BlockingRecv error: recv got EOF from 127.0.0.1:43844 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:27:12.462491 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19851
I20260618 22:27:12.466847 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19985
I20260618 22:27:12.471326 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20119
I20260618 22:27:12.475525 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 19757
2026-06-18T22:27:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2016 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[20260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[20260](info): set up 2 sockets
Jun 18 22:27:12 dist-test-slave-jm5g krb5kdc[20260](info): commencing operation
krb5kdc: starting...
W20260618 22:27:13.434460 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:13.553848 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44135
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:13.662233 20273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:13.662542 20273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:13.662629 20273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:13.666785 20273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:13.666853 20273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:13.666880 20273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:13.666900 20273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:13.666929 20273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:13.672261 20273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:44135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:13.673332 20273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:13.674340 20273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:13.682716 20273 server_base.cc:1034] running on GCE node
W20260618 22:27:13.682673 20281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:13.682668 20278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:13.682668 20279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:13.683504 20273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:13.684641 20273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:13.685819 20273 hybrid_clock.cc:648] HybridClock initialized: now 1781821633685799 us; error 40 us; skew 500 ppm
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:13.693950 20273 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:13.694975 20273 webserver.cc:460] Webserver started at http://127.16.243.190:34013/ using document root <none> and password file <none>
I20260618 22:27:13.695518 20273 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:13.695576 20273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:13.695742 20273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:13.697786 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "ed22750912fa4508a7e5f7641f2baf40"
format_stamp: "Formatted at 2026-06-18 22:27:13 on dist-test-slave-jm5g"
server_key: "68822d89ffadc8156a131c7754ac4091"
server_key_iv: "dae56a536f7ace70d5201611c40b9cbd"
server_key_version: "encryptionkey@0"
I20260618 22:27:13.698307 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "ed22750912fa4508a7e5f7641f2baf40"
format_stamp: "Formatted at 2026-06-18 22:27:13 on dist-test-slave-jm5g"
server_key: "68822d89ffadc8156a131c7754ac4091"
server_key_iv: "dae56a536f7ace70d5201611c40b9cbd"
server_key_version: "encryptionkey@0"
I20260618 22:27:13.702039 20273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:13.704632 20288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:13.705897 20273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:13.706002 20273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "ed22750912fa4508a7e5f7641f2baf40"
format_stamp: "Formatted at 2026-06-18 22:27:13 on dist-test-slave-jm5g"
server_key: "68822d89ffadc8156a131c7754ac4091"
server_key_iv: "dae56a536f7ace70d5201611c40b9cbd"
server_key_version: "encryptionkey@0"
I20260618 22:27:13.706135 20273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:13.729869 20273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:13.733289 20273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:13.733464 20273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:13.740459 20273 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:44135
I20260618 22:27:13.740713 20340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:44135 every 8 connection(s)
I20260618 22:27:13.741457 20273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:13.744328 20341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:13.749763 20341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40: Bootstrap starting.
I20260618 22:27:13.750150 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20273
I20260618 22:27:13.750255 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:13.750478 17358 external_mini_cluster.cc:1428] Setting key 42a807a3d587e23f4039365d7e866abb
I20260618 22:27:13.752189 20341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:13.752892 20341 log.cc:824] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:13.754935 20341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40: No bootstrap required, opened a new log
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:13.759042 20341 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } }
I20260618 22:27:13.759258 20341 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:13.759317 20341 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed22750912fa4508a7e5f7641f2baf40, State: Initialized, Role: FOLLOWER
I20260618 22:27:13.759729 20341 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } }
I20260618 22:27:13.759848 20341 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:13.759912 20341 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:13.760010 20341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:13.760984 20341 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } }
I20260618 22:27:13.761293 20341 leader_election.cc:302] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed22750912fa4508a7e5f7641f2baf40; no voters:
I20260618 22:27:13.761623 20341 leader_election.cc:288] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:13.761679 20346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:13.761904 20346 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [term 1 LEADER]: Becoming Leader. State: Replica: ed22750912fa4508a7e5f7641f2baf40, State: Running, Role: LEADER
I20260618 22:27:13.762270 20346 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } }
I20260618 22:27:13.762923 20341 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:13.764076 20348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed22750912fa4508a7e5f7641f2baf40. Latest consensus state: current_term: 1 leader_uuid: "ed22750912fa4508a7e5f7641f2baf40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } } }
I20260618 22:27:13.764102 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed22750912fa4508a7e5f7641f2baf40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed22750912fa4508a7e5f7641f2baf40" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44135 } } }
I20260618 22:27:13.764282 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:13.764274 20348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:13.764607 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:13.751788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.16.243.190:44135)
0618 22:27:13.752184 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:13.752200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:13.752235 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:27:13.752878 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:13.752894 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:13.753279 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:13.753632 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:13.754438 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:13.755447 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:13.756574 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:13.757030 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:13.757137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:13.759764 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:13.759768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:13.759781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:13.759808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:13.761386 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:13.761852 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:13.761858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:13.761864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:13.762010 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:13.762271 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:13.762274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:13.762276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:13.762836 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:13.763031 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:13.763284 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 22:27:13.763609 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260618 22:27:13.764680 20355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:13.767797 20355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:13.767848 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:27:13.773082 20355 catalog_manager.cc:1347] Generated new cluster ID: 369e75fced0e478f9030ac83d3bb8776
I20260618 22:27:13.773149 20355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:13.784228 20355 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:13.785180 20355 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:13.793812 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:13.788171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.16.243.190:44135)
0618 22:27:13.788282 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:13.788286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:13.788537 (+ 251us) server_negotiation.cc:407] Connection header received
0618 22:27:13.788678 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:13.788681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:13.788733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:13.788812 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:13.789393 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:13.790173 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:13.791390 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:13.791746 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:13.791844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:13.792751 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:13.792755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:13.792758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:13.793339 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:13.793384 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:13.793406 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:13.793497 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:13.793588 (+ 91us) server_negotiation.cc:299] Negotiation successful
0618 22:27:13.793654 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:13.800755 20355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed22750912fa4508a7e5f7641f2baf40: Generated new TSK 0
I20260618 22:27:13.801448 20355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:13.808959 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20273
2026-06-18T22:27:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20374](info): set up 2 sockets
Jun 18 22:27:13 dist-test-slave-jm5g krb5kdc[20374](info): commencing operation
krb5kdc: starting...
W20260618 22:27:14.758262 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:14 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:14.874737 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:32855
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:32855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:14.983366 20387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:14.983695 20387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:14.983754 20387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:14.987989 20387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:14.988070 20387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:14.988096 20387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:14.988118 20387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:14.988139 20387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:14.993707 20387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:32855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:32855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:14.994891 20387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:14.995927 20387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:15.004539 20392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.004552 20393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.004522 20395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:15.004705 20387 server_base.cc:1034] running on GCE node
I20260618 22:27:15.005579 20387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:15.006816 20387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:15.007999 20387 hybrid_clock.cc:648] HybridClock initialized: now 1781821635007984 us; error 42 us; skew 500 ppm
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:15.010983 20387 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:15.011905 20387 webserver.cc:460] Webserver started at http://127.16.243.190:46351/ using document root <none> and password file <none>
I20260618 22:27:15.012434 20387 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:15.012491 20387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:15.012668 20387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:15.014765 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "5e19639692784003915dd00f08cdf6d0"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "ec77c24eb76a89c2eca20a4a4abc95cd"
server_key_iv: "dfedc730d4f631f9c5f8e8d6ee03ae23"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.015300 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "5e19639692784003915dd00f08cdf6d0"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "ec77c24eb76a89c2eca20a4a4abc95cd"
server_key_iv: "dfedc730d4f631f9c5f8e8d6ee03ae23"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.019073 20387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:27:15.021612 20402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.022955 20387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:15.023097 20387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "5e19639692784003915dd00f08cdf6d0"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "ec77c24eb76a89c2eca20a4a4abc95cd"
server_key_iv: "dfedc730d4f631f9c5f8e8d6ee03ae23"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.023203 20387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:15.047606 20387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:15.050710 20387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:15.050902 20387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:15.057797 20387 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:32855
I20260618 22:27:15.058060 20454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:32855 every 8 connection(s)
I20260618 22:27:15.058842 20387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:15.061120 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20387
I20260618 22:27:15.061231 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:15.061480 17358 external_mini_cluster.cc:1428] Setting key c65de8649d40a3e8c68820606096bfe7
I20260618 22:27:15.061658 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:15.067988 20455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0: Bootstrap starting.
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:15.070308 20455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:15.071198 20455 log.cc:824] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:15.073415 20455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0: No bootstrap required, opened a new log
I20260618 22:27:15.075563 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.062860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.16.243.190:32855)
0618 22:27:15.063315 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:15.063334 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:15.063371 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:27:15.064381 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:15.064401 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:15.064759 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:15.065068 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:15.065864 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.066918 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.067768 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.068278 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.068419 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.070945 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:15.070951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:15.070965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:15.071000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:15.072659 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.073170 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.073175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.073180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.073293 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.073591 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.073594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.073596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.074021 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:15.074179 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:15.074466 (+ 287us) server_negotiation.cc:299] Negotiation successful
0618 22:27:15.074797 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":139,"threads_started":1}
I20260618 22:27:15.077406 20455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } }
I20260618 22:27:15.077669 20455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:15.077768 20455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e19639692784003915dd00f08cdf6d0, State: Initialized, Role: FOLLOWER
I20260618 22:27:15.078186 20455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } }
I20260618 22:27:15.078315 20455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:15.078372 20455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:15.078461 20455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:15.079387 20455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } }
I20260618 22:27:15.079752 20455 leader_election.cc:302] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e19639692784003915dd00f08cdf6d0; no voters:
I20260618 22:27:15.080071 20455 leader_election.cc:288] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:15.080399 20461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:15.080638 20461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5e19639692784003915dd00f08cdf6d0, State: Running, Role: LEADER
I20260618 22:27:15.080976 20461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } }
I20260618 22:27:15.081372 20455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:15.082607 20462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e19639692784003915dd00f08cdf6d0. Latest consensus state: current_term: 1 leader_uuid: "5e19639692784003915dd00f08cdf6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } } }
I20260618 22:27:15.082734 20462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:15.083082 20460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e19639692784003915dd00f08cdf6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e19639692784003915dd00f08cdf6d0" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 32855 } } }
I20260618 22:27:15.083164 20469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:15.083164 20460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:15.086359 20469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:15.092010 20469 catalog_manager.cc:1347] Generated new cluster ID: 32d3747b1a0f44bf8f6ff761a22eb5c9
I20260618 22:27:15.092108 20469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:15.100416 20469 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:15.101727 20469 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:15.114775 20469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e19639692784003915dd00f08cdf6d0: Generated new TSK 0
I20260618 22:27:15.115649 20469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:15.145565 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:15.256338 20481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:15.256611 20481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:15.256654 20481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:15.260653 20481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:15.260720 20481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:15.260787 20481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:15.266086 20481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:15.267166 20481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:15.268177 20481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:15.277287 20486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.277300 20489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.277300 20487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:15.277624 20481 server_base.cc:1034] running on GCE node
I20260618 22:27:15.278038 20481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:15.278683 20481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:15.279850 20481 hybrid_clock.cc:648] HybridClock initialized: now 1781821635279830 us; error 36 us; skew 500 ppm
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:15.282603 20481 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:15.283497 20481 webserver.cc:460] Webserver started at http://127.16.243.129:46443/ using document root <none> and password file <none>
I20260618 22:27:15.284085 20481 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:15.284152 20481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:15.284341 20481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:15.286322 20481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "3743406f8a1949c088127e4e5a863118"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "a486329f0bba681b036518b5ccbb14b9"
server_key_iv: "a5f2f6a71fe57d998888c7b1e158524b"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.286885 20481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "3743406f8a1949c088127e4e5a863118"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "a486329f0bba681b036518b5ccbb14b9"
server_key_iv: "a5f2f6a71fe57d998888c7b1e158524b"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.290624 20481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:15.293013 20496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.294241 20481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:15.294354 20481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "3743406f8a1949c088127e4e5a863118"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "a486329f0bba681b036518b5ccbb14b9"
server_key_iv: "a5f2f6a71fe57d998888c7b1e158524b"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.294447 20481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:15.312882 20481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:15.315954 20481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:15.316143 20481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:15.316735 20481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:15.317581 20481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:15.317629 20481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.317682 20481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:15.317711 20481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.326509 20481 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:41735
I20260618 22:27:15.326532 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:41735 every 8 connection(s)
I20260618 22:27:15.327454 20481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:15.332122 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20481
I20260618 22:27:15.332243 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:15.332504 17358 external_mini_cluster.cc:1428] Setting key 8eac18b521904231294f329fe6913e93
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:15.341133 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.329236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:39439 (local address 127.16.243.190:32855)
0618 22:27:15.329357 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:15.329360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:15.330231 (+ 871us) server_negotiation.cc:407] Connection header received
0618 22:27:15.331597 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:15.331600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:15.331656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:15.331727 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:15.332971 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.333638 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.334790 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.335033 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.335126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.337643 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:15.337647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:15.337649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:15.337672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:15.339143 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.339658 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.339661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.339662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.339738 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.340044 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.340047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.340049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.340235 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:15.340320 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:15.340826 (+ 506us) server_negotiation.cc:299] Negotiation successful
0618 22:27:15.340955 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:15.341958 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.329591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:32855 (local address 127.16.243.129:39439)
0618 22:27:15.330067 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:15.330110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:15.331376 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:15.331933 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:15.331940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:15.332319 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:15.332834 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.332847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.333749 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.333751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.334685 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.334695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.335197 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.335202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.335387 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.335981 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:15.335996 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:15.337483 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:15.339258 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.339264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.339282 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.339536 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.339821 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.339824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.339826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.339963 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.340330 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:15.340337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:15.340604 (+ 267us) client_negotiation.cc:769] Sending connection context
0618 22:27:15.340778 (+ 174us) client_negotiation.cc:240] Negotiation successful
0618 22:27:15.341113 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260618 22:27:15.343206 20610 heartbeater.cc:347] Connected to a master server at 127.16.243.190:32855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:15.343433 20610 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:15.343931 20610 heartbeater.cc:511] Master 127.16.243.190:32855 requested a full tablet report, sending...
I20260618 22:27:15.345448 20419 ts_manager.cc:194] Registered new tserver with Master: 3743406f8a1949c088127e4e5a863118 (127.16.243.129:41735)
I20260618 22:27:15.347016 20419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:39439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:15.357738 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:15.475394 20615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:15.475698 20615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:15.475746 20615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:15.480556 20615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:15.480649 20615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:15.480715 20615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:15.487712 20615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:15.489017 20615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:15.490131 20615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:15.500464 20621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.500566 20620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.500478 20623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:15.500841 20615 server_base.cc:1034] running on GCE node
I20260618 22:27:15.501295 20615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:15.502003 20615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:15.503193 20615 hybrid_clock.cc:648] HybridClock initialized: now 1781821635503169 us; error 45 us; skew 500 ppm
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:15.506143 20615 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:15.507162 20615 webserver.cc:460] Webserver started at http://127.16.243.130:37919/ using document root <none> and password file <none>
I20260618 22:27:15.507709 20615 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:15.507769 20615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:15.507938 20615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:15.510063 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "6e7cac7ccc9242e488a5f2b8f8d6dfd3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "8eb9c49208d0a7beda13b14406561e2d"
server_key_iv: "4940744fee944bba66d998f495119213"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.510687 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "6e7cac7ccc9242e488a5f2b8f8d6dfd3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "8eb9c49208d0a7beda13b14406561e2d"
server_key_iv: "4940744fee944bba66d998f495119213"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.514703 20615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:15.517568 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.519090 20615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:15.519214 20615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "6e7cac7ccc9242e488a5f2b8f8d6dfd3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "8eb9c49208d0a7beda13b14406561e2d"
server_key_iv: "4940744fee944bba66d998f495119213"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.519326 20615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:15.533066 20615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:15.536262 20615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:15.536419 20615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:15.536969 20615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:15.537768 20615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:15.537814 20615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.537864 20615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:15.537885 20615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.546519 20615 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:38799
I20260618 22:27:15.546594 20743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:38799 every 8 connection(s)
I20260618 22:27:15.547485 20615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:15.554536 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20615
I20260618 22:27:15.554685 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:15.554958 17358 external_mini_cluster.cc:1428] Setting key a493eeb822fa8d94f0399b6e2c7c3407
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:15.562331 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.549348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:45959 (local address 127.16.243.190:32855)
0618 22:27:15.549494 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:15.549499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:15.550441 (+ 942us) server_negotiation.cc:407] Connection header received
0618 22:27:15.551810 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:15.551812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:15.551861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:15.551936 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:15.553251 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.553783 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.555290 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.555523 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.555612 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.558228 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:15.558232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:15.558235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:15.558265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:15.559806 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.560483 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.560487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.560489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.560597 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.561027 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.561031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.561033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.561310 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:15.561429 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:15.562013 (+ 584us) server_negotiation.cc:299] Negotiation successful
0618 22:27:15.562156 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:27:15.563190 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.549763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:32855 (local address 127.16.243.130:45959)
0618 22:27:15.550275 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:15.550321 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:15.551589 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:15.552150 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:15.552157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:15.552535 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:15.553085 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.553100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.553901 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.553904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.555169 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.555180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.555675 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.555680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.555837 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.556431 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:15.556456 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:15.558039 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:15.559943 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.560030 (+ 87us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.560059 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.560372 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.560718 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.560721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.560723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.560895 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.561458 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:15.561464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:15.561739 (+ 275us) client_negotiation.cc:769] Sending connection context
0618 22:27:15.561928 (+ 189us) client_negotiation.cc:240] Negotiation successful
0618 22:27:15.562222 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:15.564565 20744 heartbeater.cc:347] Connected to a master server at 127.16.243.190:32855
I20260618 22:27:15.564824 20744 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:15.565392 20744 heartbeater.cc:511] Master 127.16.243.190:32855 requested a full tablet report, sending...
I20260618 22:27:15.566705 20419 ts_manager.cc:194] Registered new tserver with Master: 6e7cac7ccc9242e488a5f2b8f8d6dfd3 (127.16.243.130:38799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:15.567323 20419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:45959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:15.585395 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:15.698052 20749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:15.698321 20749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:15.698366 20749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:15.702491 20749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:15.702586 20749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:15.702673 20749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:15.708070 20749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:15.709122 20749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:15.710127 20749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:15.719376 20757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:15.719475 20749 server_base.cc:1034] running on GCE node
W20260618 22:27:15.719383 20754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:15.719416 20755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:15.720057 20749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:15.720696 20749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:15.721863 20749 hybrid_clock.cc:648] HybridClock initialized: now 1781821635721842 us; error 35 us; skew 500 ppm
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:15.724632 20749 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:15.725502 20749 webserver.cc:460] Webserver started at http://127.16.243.131:37071/ using document root <none> and password file <none>
I20260618 22:27:15.726017 20749 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:15.726079 20749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:15.726275 20749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:15.728372 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "0ba1192c30c441c48d938241cc2586a3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "e4d7e69e7b8755f8d3ce8cc96ff5ffc6"
server_key_iv: "66e5c6ebf9cbdd857f2c7f4f2b577542"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.728967 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "0ba1192c30c441c48d938241cc2586a3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "e4d7e69e7b8755f8d3ce8cc96ff5ffc6"
server_key_iv: "66e5c6ebf9cbdd857f2c7f4f2b577542"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.732746 20749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:27:15.735308 20764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.736539 20749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:15.736644 20749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "0ba1192c30c441c48d938241cc2586a3"
format_stamp: "Formatted at 2026-06-18 22:27:15 on dist-test-slave-jm5g"
server_key: "e4d7e69e7b8755f8d3ce8cc96ff5ffc6"
server_key_iv: "66e5c6ebf9cbdd857f2c7f4f2b577542"
server_key_version: "encryptionkey@0"
I20260618 22:27:15.736758 20749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:15.749081 20749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:15.752123 20749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:15.752287 20749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:15.752859 20749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:15.753649 20749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:15.753692 20749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.753733 20749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:15.753762 20749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:15.762491 20749 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:34247
I20260618 22:27:15.762537 20877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:34247 every 8 connection(s)
I20260618 22:27:15.763440 20749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:15.771844 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20749
I20260618 22:27:15.771973 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:15.772272 17358 external_mini_cluster.cc:1428] Setting key cefdccb451ad7fd2f9e4a6e345dfd5ec
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:15.776929 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.765232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:47907 (local address 127.16.243.190:32855)
0618 22:27:15.765396 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:15.765401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:15.766273 (+ 872us) server_negotiation.cc:407] Connection header received
0618 22:27:15.767635 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:15.767639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:15.767689 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:15.767767 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:15.768975 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.769508 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.770782 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.771014 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.771099 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.773754 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:15.773757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:15.773759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:15.773783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:15.775184 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.775675 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.775678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.775679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.775752 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.776017 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.776019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.776020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.776189 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:15.776259 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:15.776709 (+ 450us) server_negotiation.cc:299] Negotiation successful
0618 22:27:15.776811 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:27:15.777776 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.765631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:32855 (local address 127.16.243.131:47907)
0618 22:27:15.766116 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:15.766163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:15.767420 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:15.767948 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:15.767955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:15.768323 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:15.768817 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.768831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.769645 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.769648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.770653 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:15.770663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.771140 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.771143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:15.771289 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.771885 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:15.771906 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:15.773609 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:15.775291 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.775297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.775312 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.775598 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.775813 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:15.775815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:15.775816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:15.775944 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:15.776258 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:15.776262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:15.776522 (+ 260us) client_negotiation.cc:769] Sending connection context
0618 22:27:15.776680 (+ 158us) client_negotiation.cc:240] Negotiation successful
0618 22:27:15.776947 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260618 22:27:15.778890 20878 heartbeater.cc:347] Connected to a master server at 127.16.243.190:32855
I20260618 22:27:15.779136 20878 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:15.779623 20878 heartbeater.cc:511] Master 127.16.243.190:32855 requested a full tablet report, sending...
I20260618 22:27:15.780658 20419 ts_manager.cc:194] Registered new tserver with Master: 0ba1192c30c441c48d938241cc2586a3 (127.16.243.131:34247)
I20260618 22:27:15.781281 20419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:47907
I20260618 22:27:15.786269 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821635, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:15.810878 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:15.802530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.16.243.190:32855)
0618 22:27:15.802702 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:15.802707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:15.802851 (+ 144us) server_negotiation.cc:407] Connection header received
0618 22:27:15.803022 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:15.803025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:15.803071 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:15.803145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:15.803951 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.804475 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.805298 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:15.805525 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:15.805614 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:15.808069 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:15.808073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:15.808075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:15.808103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:15.809268 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.809669 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.809673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.809675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.809754 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:15.809990 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:15.809994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:15.809995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:15.810227 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:15.810342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:15.810595 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 22:27:15.810722 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260618 22:27:15.811357 20419 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55290
I20260618 22:27:15.812462 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20481
I20260618 22:27:15.816727 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20615
I20260618 22:27:15.821079 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20749
I20260618 22:27:15.825474 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20387
2026-06-18T22:27:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2018 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20890](info): set up 2 sockets
Jun 18 22:27:15 dist-test-slave-jm5g krb5kdc[20890](info): commencing operation
krb5kdc: starting...
W20260618 22:27:16.786300 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:16 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:16.899639 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42811
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:17.008437 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:17.008724 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:17.008782 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:17.012835 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:17.012908 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:17.012935 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:17.012959 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:17.012977 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:17.018404 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:17.019497 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:17.020551 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:17.028810 20903 server_base.cc:1034] running on GCE node
W20260618 22:27:17.028734 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.028755 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.028734 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.029520 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:17.030680 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:17.031831 20903 hybrid_clock.cc:648] HybridClock initialized: now 1781821637031808 us; error 51 us; skew 500 ppm
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:17.034616 20903 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:17.035492 20903 webserver.cc:460] Webserver started at http://127.16.243.190:34831/ using document root <none> and password file <none>
I20260618 22:27:17.036013 20903 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:17.036077 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:17.036262 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:17.038280 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "198094e1109a4ed9aedb01fe6be6394a"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "716c70ec83cd2e2616d4005272f034fb"
server_key_iv: "a48014fcc9342be19238a6c798d25012"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.038844 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "198094e1109a4ed9aedb01fe6be6394a"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "716c70ec83cd2e2616d4005272f034fb"
server_key_iv: "a48014fcc9342be19238a6c798d25012"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.042471 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:17.044950 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.046243 20903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:17.046365 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "198094e1109a4ed9aedb01fe6be6394a"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "716c70ec83cd2e2616d4005272f034fb"
server_key_iv: "a48014fcc9342be19238a6c798d25012"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.046478 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:17.079851 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:17.083120 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:17.083300 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:17.090199 20903 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:42811
I20260618 22:27:17.090206 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:42811 every 8 connection(s)
I20260618 22:27:17.091161 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:17.093942 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:17.095907 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20903
I20260618 22:27:17.096050 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:17.096295 17358 external_mini_cluster.cc:1428] Setting key 5b465ac6a9e7040c3cfe2a7858da1ed1
I20260618 22:27:17.101085 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a: Bootstrap starting.
I20260618 22:27:17.103790 20971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:17.104748 20971 log.cc:824] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:17.107055 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a: No bootstrap required, opened a new log
I20260618 22:27:17.110589 20971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } }
I20260618 22:27:17.110850 20971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:17.110922 20971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198094e1109a4ed9aedb01fe6be6394a, State: Initialized, Role: FOLLOWER
I20260618 22:27:17.111229 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.097774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.16.243.190:42811)
0618 22:27:17.098252 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.098281 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.098326 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:27:17.099117 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.099145 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.099611 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.100026 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.100687 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.102153 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.103132 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.103821 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.103989 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.106646 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.106653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.106670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.106701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.108385 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.108816 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.108822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.108827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.108938 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.109207 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.109211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.109212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.109605 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.109800 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.110089 (+ 289us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.110372 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260618 22:27:17.111395 20971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } }
I20260618 22:27:17.111552 20971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:17.111617 20971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:17.111712 20971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:17.112792 20971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } }
I20260618 22:27:17.113161 20971 leader_election.cc:302] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198094e1109a4ed9aedb01fe6be6394a; no voters:
I20260618 22:27:17.113492 20971 leader_election.cc:288] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:17.113605 20976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:17.113832 20976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [term 1 LEADER]: Becoming Leader. State: Replica: 198094e1109a4ed9aedb01fe6be6394a, State: Running, Role: LEADER
I20260618 22:27:17.114114 20976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } }
I20260618 22:27:17.114840 20971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:17.115751 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198094e1109a4ed9aedb01fe6be6394a. Latest consensus state: current_term: 1 leader_uuid: "198094e1109a4ed9aedb01fe6be6394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } } }
I20260618 22:27:17.115775 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198094e1109a4ed9aedb01fe6be6394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198094e1109a4ed9aedb01fe6be6394a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42811 } } }
I20260618 22:27:17.115969 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:17.116022 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:17.116405 20985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:17.119496 20985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:17.125078 20985 catalog_manager.cc:1347] Generated new cluster ID: f32c56ed50634d41b3a41b4cce5daea8
I20260618 22:27:17.125150 20985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:17.157218 20985 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:17.158216 20985 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:17.164628 20985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 198094e1109a4ed9aedb01fe6be6394a: Generated new TSK 0
I20260618 22:27:17.165372 20985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:17.196861 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:17.308926 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:17.309217 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:17.309262 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:17.313361 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:17.313426 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:17.313488 20997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:17.319053 20997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:17.320091 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:17.321043 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:17.330260 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.330269 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.330265 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.330577 20997 server_base.cc:1034] running on GCE node
I20260618 22:27:17.330936 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:17.331532 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:17.332685 20997 hybrid_clock.cc:648] HybridClock initialized: now 1781821637332666 us; error 30 us; skew 500 ppm
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:17.335491 20997 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:17.336345 20997 webserver.cc:460] Webserver started at http://127.16.243.129:37523/ using document root <none> and password file <none>
I20260618 22:27:17.336861 20997 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:17.336917 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:17.337080 20997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:17.339139 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "768874450b6e47399674d6f3c865d7d6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "06810f26bc37e43fc4947cddf92615e4"
server_key_iv: "4f4f2a75a535d7abdd38ef44537402a8"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.339632 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "768874450b6e47399674d6f3c865d7d6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "06810f26bc37e43fc4947cddf92615e4"
server_key_iv: "4f4f2a75a535d7abdd38ef44537402a8"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.343179 20997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:17.345686 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.346988 20997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:17.347091 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "768874450b6e47399674d6f3c865d7d6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "06810f26bc37e43fc4947cddf92615e4"
server_key_iv: "4f4f2a75a535d7abdd38ef44537402a8"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.347184 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:17.378315 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:17.381520 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:17.381683 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:17.382229 20997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:17.383072 20997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:17.383121 20997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.383165 20997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:17.383186 20997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.391425 20997 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:38405
I20260618 22:27:17.391475 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:38405 every 8 connection(s)
I20260618 22:27:17.392287 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:17.393110 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 20997
I20260618 22:27:17.393213 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:17.393472 17358 external_mini_cluster.cc:1428] Setting key 2cab250c961dce15eebe56f7d30c3fce
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:17.405920 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.394106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:42033 (local address 127.16.243.190:42811)
0618 22:27:17.394237 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.394240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.395176 (+ 936us) server_negotiation.cc:407] Connection header received
0618 22:27:17.396506 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.396510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.396563 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.396635 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.397787 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.398323 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.399462 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.399684 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.399769 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.402096 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.402099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.402101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.402127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.403595 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.404161 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.404165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.404167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.404258 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.404670 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.404673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.404674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.404866 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.404950 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.405610 (+ 660us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.405748 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:27:17.406785 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.394496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42811 (local address 127.16.243.129:42033)
0618 22:27:17.395040 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:17.395085 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:17.396304 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:17.396808 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:17.396816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:17.397166 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:17.397651 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.397664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.398438 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.398440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.399351 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.399360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.399820 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.399824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.399977 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.400532 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:17.400548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:17.401915 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:17.403724 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.403729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.403743 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.404042 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.404347 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.404350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.404352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.404544 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.404960 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:17.404967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:17.405305 (+ 338us) client_negotiation.cc:769] Sending connection context
0618 22:27:17.405529 (+ 224us) client_negotiation.cc:240] Negotiation successful
0618 22:27:17.405847 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":116,"threads_started":1}
I20260618 22:27:17.407982 21126 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42811
I20260618 22:27:17.408224 21126 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:17.408720 21126 heartbeater.cc:511] Master 127.16.243.190:42811 requested a full tablet report, sending...
I20260618 22:27:17.410214 20935 ts_manager.cc:194] Registered new tserver with Master: 768874450b6e47399674d6f3c865d7d6 (127.16.243.129:38405)
I20260618 22:27:17.411361 20935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:42033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:17.419339 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:17.532218 21131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:17.532513 21131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:17.532559 21131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:17.536880 21131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:17.536947 21131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:17.537014 21131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:17.542514 21131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:17.543659 21131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:17.544834 21131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:17.554404 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.554437 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.554581 21131 server_base.cc:1034] running on GCE node
W20260618 22:27:17.554436 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.555089 21131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:17.555733 21131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:17.556890 21131 hybrid_clock.cc:648] HybridClock initialized: now 1781821637556884 us; error 33 us; skew 500 ppm
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:17.559603 21131 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:17.560428 21131 webserver.cc:460] Webserver started at http://127.16.243.130:43615/ using document root <none> and password file <none>
I20260618 22:27:17.560942 21131 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:17.560998 21131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:17.561159 21131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:17.563133 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "ffb89ecdeb294538ba0f67d35ae32738"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "7e30e50fd028c169e8e920dbf7d40ea2"
server_key_iv: "9287d25664f28d784751be8be1864c1f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.563618 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb89ecdeb294538ba0f67d35ae32738"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "7e30e50fd028c169e8e920dbf7d40ea2"
server_key_iv: "9287d25664f28d784751be8be1864c1f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.567199 21131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:17.569620 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.570933 21131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:17.571051 21131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "ffb89ecdeb294538ba0f67d35ae32738"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "7e30e50fd028c169e8e920dbf7d40ea2"
server_key_iv: "9287d25664f28d784751be8be1864c1f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.571154 21131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:17.592361 21131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:17.595638 21131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:17.595806 21131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:17.596395 21131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:17.597210 21131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:17.597255 21131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.597294 21131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:17.597311 21131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.605544 21131 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:43009
I20260618 22:27:17.605579 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:43009 every 8 connection(s)
I20260618 22:27:17.606458 21131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:17.616115 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21131
I20260618 22:27:17.616216 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:17.616533 17358 external_mini_cluster.cc:1428] Setting key 541acf25fa02eb43c2c30af1ddfe2488
I20260618 22:27:17.619858 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.608277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:47059 (local address 127.16.243.190:42811)
0618 22:27:17.608416 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.608420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.609305 (+ 885us) server_negotiation.cc:407] Connection header received
0618 22:27:17.610692 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.610696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.610752 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.610825 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.611993 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.612526 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.613639 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.613859 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.613945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.616462 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.616467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.616470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.616497 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.617911 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.618387 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.618391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.618393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.618475 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.618787 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.618789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.618791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.619002 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.619091 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.619582 (+ 491us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.619701 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:27:17.620720 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.608658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42811 (local address 127.16.243.130:47059)
0618 22:27:17.609157 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:17.609201 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:17.610464 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:17.610991 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:17.610998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:17.611372 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:17.611853 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.611866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.612644 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.612646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.613524 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.613533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.613996 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.614000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.614160 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.614735 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:17.614751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:17.616268 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:17.618036 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.618041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.618053 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.618290 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.618575 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.618578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.618579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.618712 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.619102 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:17.619107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:17.619335 (+ 228us) client_negotiation.cc:769] Sending connection context
0618 22:27:17.619495 (+ 160us) client_negotiation.cc:240] Negotiation successful
0618 22:27:17.619819 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260618 22:27:17.621798 21260 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42811
I20260618 22:27:17.622052 21260 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:17.622514 21260 heartbeater.cc:511] Master 127.16.243.190:42811 requested a full tablet report, sending...
I20260618 22:27:17.623577 20935 ts_manager.cc:194] Registered new tserver with Master: ffb89ecdeb294538ba0f67d35ae32738 (127.16.243.130:43009)
I20260618 22:27:17.624137 20935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:47059
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:17.643321 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:17.751711 21265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:17.751962 21265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:17.752007 21265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:17.755966 21265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:17.756021 21265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:17.756078 21265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:17.761178 21265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:17.762144 21265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:17.763152 21265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:17.772220 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:17.772213 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.772307 21265 server_base.cc:1034] running on GCE node
W20260618 22:27:17.772231 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:17.772800 21265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:17.773404 21265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:17.774581 21265 hybrid_clock.cc:648] HybridClock initialized: now 1781821637774527 us; error 38 us; skew 500 ppm
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:17.777199 21265 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:17.778007 21265 webserver.cc:460] Webserver started at http://127.16.243.131:36741/ using document root <none> and password file <none>
I20260618 22:27:17.778518 21265 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:17.778625 21265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:17.778824 21265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:17.781278 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "83db77b444fc4300a50c167480ecd5c6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "9806e791610ea2fd0facd144685b18f7"
server_key_iv: "b7cf6d52e3561d9528edaac96860729f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.781838 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "83db77b444fc4300a50c167480ecd5c6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "9806e791610ea2fd0facd144685b18f7"
server_key_iv: "b7cf6d52e3561d9528edaac96860729f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.785527 21265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:17.788000 21280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.789248 21265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:17.789357 21265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "83db77b444fc4300a50c167480ecd5c6"
format_stamp: "Formatted at 2026-06-18 22:27:17 on dist-test-slave-jm5g"
server_key: "9806e791610ea2fd0facd144685b18f7"
server_key_iv: "b7cf6d52e3561d9528edaac96860729f"
server_key_version: "encryptionkey@0"
I20260618 22:27:17.789453 21265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:17.833282 21265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:17.836508 21265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:17.836681 21265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:17.837262 21265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:17.838069 21265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:17.838115 21265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.838167 21265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:17.838192 21265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:17.846499 21265 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:38761
I20260618 22:27:17.846537 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:38761 every 8 connection(s)
I20260618 22:27:17.847393 21265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:17.849517 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21265
I20260618 22:27:17.849650 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:17.849913 17358 external_mini_cluster.cc:1428] Setting key b22ccdbb4b2488d72586fb6e427132dd
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:17.860543 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.849175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:55357 (local address 127.16.243.190:42811)
0618 22:27:17.849317 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.849321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.850238 (+ 917us) server_negotiation.cc:407] Connection header received
0618 22:27:17.851888 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.851891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.851945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.852014 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.853147 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.853670 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.854814 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.855050 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.855133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.857541 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.857544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.857547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.857571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.858792 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.859237 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.859240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.859241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.859307 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.859593 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.859595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.859596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.859758 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.859827 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.860302 (+ 475us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.860411 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:27:17.861316 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.849600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42811 (local address 127.16.243.131:55357)
0618 22:27:17.850089 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:17.850142 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:17.851674 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:17.852193 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:17.852200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:17.852535 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:17.853009 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.853021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.853759 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.853761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.854698 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:17.854709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.855186 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.855189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:17.855338 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.855941 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:17.855957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:17.857379 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:17.858886 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.858891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.858904 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.859165 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.859365 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:17.859367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:17.859368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:17.859517 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:17.859824 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:17.859829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:17.860055 (+ 226us) client_negotiation.cc:769] Sending connection context
0618 22:27:17.860212 (+ 157us) client_negotiation.cc:240] Negotiation successful
0618 22:27:17.860460 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260618 22:27:17.862447 21394 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42811
I20260618 22:27:17.862715 21394 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:17.863276 21394 heartbeater.cc:511] Master 127.16.243.190:42811 requested a full tablet report, sending...
I20260618 22:27:17.864269 20935 ts_manager.cc:194] Registered new tserver with Master: 83db77b444fc4300a50c167480ecd5c6 (127.16.243.131:38761)
I20260618 22:27:17.864801 20935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:55357
I20260618 22:27:17.875578 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:17.900213 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.890669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.16.243.129:38405)
0618 22:27:17.890964 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.890979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.891002 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:27:17.891070 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.891075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.891262 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.891377 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.892038 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.892879 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.893734 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.894212 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.894300 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.896736 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.896745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.896758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.896783 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.898383 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.898855 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.898860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.898862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.898952 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.899218 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.899220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.899222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.899475 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.899683 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.899888 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.900047 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:17.910856 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:17.903076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.16.243.190:42811)
0618 22:27:17.903203 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:17.903208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:17.903255 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:27:17.903370 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:17.903373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:17.903418 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:17.903495 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:17.904090 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.904605 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.905339 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:17.905561 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:17.905644 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:17.907921 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:17.907924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:17.907926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:17.907949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:17.909308 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.909764 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.909766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.909768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.909834 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:17.910055 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:17.910057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:17.910058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:17.910274 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:17.910370 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:17.910626 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:27:17.910731 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:27:17.911299 20935 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094
I20260618 22:27:17.911975 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20997
I20260618 22:27:17.916271 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21131
I20260618 22:27:17.921010 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21265
I20260618 22:27:17.925041 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 20903
2026-06-18T22:27:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2099 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[21406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[21406](info): set up 2 sockets
Jun 18 22:27:17 dist-test-slave-jm5g krb5kdc[21406](info): commencing operation
krb5kdc: starting...
W20260618 22:27:18.882367 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:18 dist-test-slave-jm5g krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:18.995776 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43447
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:19.105841 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:19.106134 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:19.106177 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:19.110275 21419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:19.110356 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:19.110375 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:19.110391 21419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:19.110405 21419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:19.115855 21419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:19.116926 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:19.117908 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:19.126040 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.126041 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.126041 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:19.126138 21419 server_base.cc:1034] running on GCE node
I20260618 22:27:19.126902 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:19.128089 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:19.129266 21419 hybrid_clock.cc:648] HybridClock initialized: now 1781821639129247 us; error 34 us; skew 500 ppm
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:19.147431 21419 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:19.148267 21419 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:19.148341 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:19.148551 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:19.150838 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "9304720014cd4b74ade84c27e66e019b"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a2ed19a400536f65279878faebceb593"
server_key_iv: "bbd986b6bbe2e471fac4f76223dcbc7d"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.151434 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "9304720014cd4b74ade84c27e66e019b"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a2ed19a400536f65279878faebceb593"
server_key_iv: "bbd986b6bbe2e471fac4f76223dcbc7d"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.155297 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:19.157861 21433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.159294 21419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:19.159410 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "9304720014cd4b74ade84c27e66e019b"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a2ed19a400536f65279878faebceb593"
server_key_iv: "bbd986b6bbe2e471fac4f76223dcbc7d"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.159523 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:19.176127 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:19.179138 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:19.179315 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:19.186307 21419 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:43447
I20260618 22:27:19.186594 21485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:43447 every 8 connection(s)
I20260618 22:27:19.186993 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:19.189836 21486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:19.192046 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21419
I20260618 22:27:19.192157 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:19.192395 17358 external_mini_cluster.cc:1428] Setting key 88c7338e2a79454f0db252d0c1e49fb9
I20260618 22:27:19.196084 21486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: Bootstrap starting.
I20260618 22:27:19.198640 21486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:19.199447 21486 log.cc:824] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:19.201777 21486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: No bootstrap required, opened a new log
I20260618 22:27:19.206416 21486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } }
I20260618 22:27:19.206756 21486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:19.206821 21486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9304720014cd4b74ade84c27e66e019b, State: Initialized, Role: FOLLOWER
I20260618 22:27:19.207310 21486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } }
I20260618 22:27:19.207460 21486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:19.207521 21486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:19.207620 21486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:19.207679 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.193848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.16.243.190:43447)
0618 22:27:19.194309 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.194324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.194362 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:27:19.195134 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.195156 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.195545 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.195891 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:19.196718 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.197708 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.198912 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.199398 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.199524 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.202265 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:19.202272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:19.202290 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:19.202323 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:19.204414 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.204880 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.204887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.204894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.205043 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.205315 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.205319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.205321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.205891 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:19.206100 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.206304 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.206658 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
I20260618 22:27:19.208593 21486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } }
I20260618 22:27:19.208994 21486 leader_election.cc:302] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9304720014cd4b74ade84c27e66e019b; no voters:
I20260618 22:27:19.209326 21486 leader_election.cc:288] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:19.209424 21491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:19.209641 21491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [term 1 LEADER]: Becoming Leader. State: Replica: 9304720014cd4b74ade84c27e66e019b, State: Running, Role: LEADER
I20260618 22:27:19.210013 21491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } }
I20260618 22:27:19.210526 21486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:19.211603 21492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9304720014cd4b74ade84c27e66e019b. Latest consensus state: current_term: 1 leader_uuid: "9304720014cd4b74ade84c27e66e019b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } } }
I20260618 22:27:19.211725 21492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:19.212093 21500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:19.212052 21493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9304720014cd4b74ade84c27e66e019b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304720014cd4b74ade84c27e66e019b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43447 } } }
I20260618 22:27:19.212284 21493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:19.215170 21500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:19.220512 21500 catalog_manager.cc:1347] Generated new cluster ID: 353beca78df048a39c684be9da93ea95
I20260618 22:27:19.220610 21500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:19.248821 21500 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:19.249832 21500 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:19.257541 21500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: Generated new TSK 0
I20260618 22:27:19.258448 21500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:19.295676 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:19.404820 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:19.405112 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:19.405159 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:19.409204 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:19.409268 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:19.409334 21512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:19.414867 21512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:19.415937 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:19.416922 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:19.426083 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.426107 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.426091 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:19.426481 21512 server_base.cc:1034] running on GCE node
I20260618 22:27:19.426971 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:19.427609 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:19.428791 21512 hybrid_clock.cc:648] HybridClock initialized: now 1781821639428773 us; error 25 us; skew 500 ppm
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:19.431660 21512 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:19.432361 21512 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:19.432430 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:19.432631 21512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:19.434680 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "996078b91e4a47079fa051758ec0ca7e"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "39314a218408696faa0714bb8d9039e8"
server_key_iv: "9f896fcc7ccb6cf24f6d471d63f5c9ae"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.435240 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "996078b91e4a47079fa051758ec0ca7e"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "39314a218408696faa0714bb8d9039e8"
server_key_iv: "9f896fcc7ccb6cf24f6d471d63f5c9ae"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.438802 21512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:19.441162 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.442350 21512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:19.442461 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "996078b91e4a47079fa051758ec0ca7e"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "39314a218408696faa0714bb8d9039e8"
server_key_iv: "9f896fcc7ccb6cf24f6d471d63f5c9ae"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.442572 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:19.459054 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:19.462204 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:19.462378 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:19.463020 21512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:19.463872 21512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:19.463924 21512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.463977 21512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:19.464005 21512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.472299 21512 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:38369
I20260618 22:27:19.472321 21639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:38369 every 8 connection(s)
I20260618 22:27:19.472855 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:19.482102 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21512
I20260618 22:27:19.482251 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:19.482492 17358 external_mini_cluster.cc:1428] Setting key 131b600bae224345802d3e91a7ba13c2
I20260618 22:27:19.485952 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.474635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37847 (local address 127.16.243.190:43447)
0618 22:27:19.474789 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.474794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.475922 (+ 1128us) server_negotiation.cc:407] Connection header received
0618 22:27:19.476717 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.476719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.476767 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.476837 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:19.477977 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.478542 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.479769 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.480002 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.480086 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.482499 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:19.482503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:19.482506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:19.482532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:19.484029 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.484502 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.484506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.484508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.484588 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.484914 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.484917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.484919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.485099 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:19.485181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.485675 (+ 494us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.485797 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:27:19.486857 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.475008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43447 (local address 127.16.243.129:37847)
0618 22:27:19.475780 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.475821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.476554 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.476994 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.477003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.477353 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:19.477831 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.477845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.478746 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.478749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.479650 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.479660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.480147 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.480149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.480295 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.480866 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:19.480882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:19.482343 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:19.484131 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.484138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.484155 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.484413 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.484666 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.484669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.484672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.484823 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.485192 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:19.485199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:19.485447 (+ 248us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.485624 (+ 177us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.485965 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260618 22:27:19.488073 21640 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43447
I20260618 22:27:19.488310 21640 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:19.488741 21640 heartbeater.cc:511] Master 127.16.243.190:43447 requested a full tablet report, sending...
I20260618 22:27:19.490159 21450 ts_manager.cc:194] Registered new tserver with Master: 996078b91e4a47079fa051758ec0ca7e (127.16.243.129:38369)
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:19.491321 21450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:37847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:19.507608 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:19.617771 21645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:19.618104 21645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:19.618182 21645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:19.622221 21645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:19.622303 21645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:19.622399 21645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:19.627822 21645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:19.628932 21645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:19.629968 21645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:19.639128 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.639128 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.639200 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:19.639221 21645 server_base.cc:1034] running on GCE node
I20260618 22:27:19.639803 21645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:19.640403 21645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:19.641575 21645 hybrid_clock.cc:648] HybridClock initialized: now 1781821639641561 us; error 32 us; skew 500 ppm
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:19.644169 21645 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:19.644824 21645 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:19.644888 21645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:19.645068 21645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:19.647102 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "cec98207ad2c42a788270d3636f13664"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "fb8d81408670aeb8670098c983aad9af"
server_key_iv: "d8966f3d52f68259cc585b2350b6e5af"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.647624 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "cec98207ad2c42a788270d3636f13664"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "fb8d81408670aeb8670098c983aad9af"
server_key_iv: "d8966f3d52f68259cc585b2350b6e5af"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.651212 21645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:27:19.653723 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.655088 21645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:19.655193 21645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "cec98207ad2c42a788270d3636f13664"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "fb8d81408670aeb8670098c983aad9af"
server_key_iv: "d8966f3d52f68259cc585b2350b6e5af"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.655288 21645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:19.664999 21645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:19.668018 21645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:19.668181 21645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:19.668725 21645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:19.669486 21645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:19.669531 21645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.669581 21645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:19.669610 21645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.677673 21645 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:34417
I20260618 22:27:19.677716 21772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:34417 every 8 connection(s)
I20260618 22:27:19.678229 21645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:19.683768 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21645
I20260618 22:27:19.683892 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:19.684197 17358 external_mini_cluster.cc:1428] Setting key d1a7ab6aac5a84924d2ab2e3a980f385
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:19.691975 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.679874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:50379 (local address 127.16.243.190:43447)
0618 22:27:19.679998 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.680002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.681098 (+ 1096us) server_negotiation.cc:407] Connection header received
0618 22:27:19.681970 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.681973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.682020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.682098 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:19.683279 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.684007 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.685332 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.685556 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.685654 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.688412 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:19.688416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:19.688419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:19.688451 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:19.689811 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.690388 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.690390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.690392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.690457 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.690835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.690838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.690839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.691009 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:19.691082 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.691745 (+ 663us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.691850 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:27:19.693145 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.680211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43447 (local address 127.16.243.130:50379)
0618 22:27:19.680953 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.680997 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.681759 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.682245 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.682252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.682645 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:19.683137 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.683150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.684144 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.684147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.685213 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.685224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.685820 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.685824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.686024 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.686675 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:19.686694 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:19.688222 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:19.689922 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.689929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.689949 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.690295 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.690533 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.690536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.690538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.690749 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.691122 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:19.691128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:19.691459 (+ 331us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.691682 (+ 223us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.692041 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:19.694253 21773 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43447
I20260618 22:27:19.694499 21773 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:19.694972 21773 heartbeater.cc:511] Master 127.16.243.190:43447 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:19.695984 21450 ts_manager.cc:194] Registered new tserver with Master: cec98207ad2c42a788270d3636f13664 (127.16.243.130:34417)
I20260618 22:27:19.696681 21450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:50379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:19.709267 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:19.815783 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:19.816054 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:19.816125 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:19.820188 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:19.820258 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:19.820343 21778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:19.825752 21778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:19.826891 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:19.827980 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:19.837445 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:19.837445 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:19.837504 21778 server_base.cc:1034] running on GCE node
W20260618 22:27:19.837477 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:19.838183 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:19.838907 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:19.840070 21778 hybrid_clock.cc:648] HybridClock initialized: now 1781821639840049 us; error 37 us; skew 500 ppm
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:19.842787 21778 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:19.843487 21778 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:19.843552 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:19.843761 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:19.845772 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "e88a7eeb369047df932acef0b2071897"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a4db74f8e99ecf55e051d573f182c007"
server_key_iv: "9196f1e216ba9df253ac8b7b306fec28"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.846287 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "e88a7eeb369047df932acef0b2071897"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a4db74f8e99ecf55e051d573f182c007"
server_key_iv: "9196f1e216ba9df253ac8b7b306fec28"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.849891 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:19.852259 21792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.853431 21778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:19.853538 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "e88a7eeb369047df932acef0b2071897"
format_stamp: "Formatted at 2026-06-18 22:27:19 on dist-test-slave-jm5g"
server_key: "a4db74f8e99ecf55e051d573f182c007"
server_key_iv: "9196f1e216ba9df253ac8b7b306fec28"
server_key_version: "encryptionkey@0"
I20260618 22:27:19.853641 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:19.866303 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:19.869252 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:19.869419 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:19.869971 21778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:19.870814 21778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:19.870860 21778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.870913 21778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:19.870939 21778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:19.879424 21905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:37919 every 8 connection(s)
I20260618 22:27:19.879421 21778 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:37919
I20260618 22:27:19.880050 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:19.885219 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21778
I20260618 22:27:19.885324 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:19.885565 17358 external_mini_cluster.cc:1428] Setting key 8ef15ed2c3b4e57fca7bff59dba8ea2d
Jun 18 22:27:19 dist-test-slave-jm5g krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821639, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:19.893121 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.881681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:44489 (local address 127.16.243.190:43447)
0618 22:27:19.881801 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.881805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.883052 (+ 1247us) server_negotiation.cc:407] Connection header received
0618 22:27:19.883916 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.883920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.883978 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.884071 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:19.885222 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.885807 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.887303 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.887616 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.887699 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.890077 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:19.890080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:19.890082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:19.890116 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:19.891394 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.891811 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.891814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.891815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.891882 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.892142 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.892144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.892145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.892308 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:19.892378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.892882 (+ 504us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.892996 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:27:19.893870 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.882045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43447 (local address 127.16.243.131:44489)
0618 22:27:19.882877 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.882919 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.883709 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.884245 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.884252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.884611 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:19.885083 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.885096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.885928 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.885931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.887194 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.887203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.887744 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.887747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.887895 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.888476 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:19.888497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:19.889926 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:19.891480 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.891485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.891497 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.891739 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.891947 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:19.891949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:19.891950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:19.892073 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:19.892412 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:19.892419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:19.892658 (+ 239us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.892820 (+ 162us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.893087 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260618 22:27:19.894941 21906 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43447
I20260618 22:27:19.895166 21906 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:19.895612 21906 heartbeater.cc:511] Master 127.16.243.190:43447 requested a full tablet report, sending...
I20260618 22:27:19.896564 21450 ts_manager.cc:194] Registered new tserver with Master: e88a7eeb369047df932acef0b2071897 (127.16.243.131:37919)
I20260618 22:27:19.897087 21450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:44489
I20260618 22:27:19.899065 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:27:19.907303 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.900773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.16.243.190:43447)
0618 22:27:19.900892 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.900896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.901075 (+ 179us) server_negotiation.cc:407] Connection header received
0618 22:27:19.901146 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.901148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.901185 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.901245 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:19.901852 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.902371 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.903189 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.903405 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.903502 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.904692 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:19.904694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:19.904696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:19.904718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:19.906043 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.906349 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.906352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.906353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.906416 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:19.906634 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:19.906636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:19.906637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:19.906826 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:19.906922 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.907071 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.907156 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:19.909885 21450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:19.913044 21450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:19.926949 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.920934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:34417 (local address 127.0.0.1:49104)
0618 22:27:19.921577 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.921590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.921684 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.922434 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.922440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.922462 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:19.922626 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.922635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.923923 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.923927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.925529 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.925536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926318 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.926322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.926421 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.926455 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.926627 (+ 172us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.926772 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":70,"thread_start_us":100,"threads_started":1}
I20260618 22:27:19.926990 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.920860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:38369 (local address 127.0.0.1:40474)
0618 22:27:19.921509 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.921560 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.921683 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.922958 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.922961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.922978 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:19.923073 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.923079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.924558 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.924565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.925560 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.925569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926593 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.926597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.926690 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.926722 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.926771 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.926836 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260618 22:27:19.926997 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.921659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.16.243.131:37919)
0618 22:27:19.922078 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.922092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.922307 (+ 215us) server_negotiation.cc:407] Connection header received
0618 22:27:19.922390 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.922395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.922627 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.922771 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:19.922949 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.924106 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.925632 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926445 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.926594 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.926719 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.926769 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.926854 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":91,"threads_started":1}
I20260618 22:27:19.927018 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.921649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.16.243.130:34417)
0618 22:27:19.921949 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.921960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.921983 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:27:19.922056 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.922061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.922268 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.922774 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:19.922918 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.923806 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.925668 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926213 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.926320 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.926457 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.926790 (+ 333us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.926886 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260618 22:27:19.927587 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.921031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.16.243.129:38369)
0618 22:27:19.922522 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.922532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.922571 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:27:19.922638 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.922644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.922809 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.923211 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:19.923379 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.924415 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.925686 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926454 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.927269 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.927371 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.927408 (+ 37us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.927474 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1396,"thread_start_us":132,"threads_started":1}
I20260618 22:27:19.927716 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.921137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:37919 (local address 127.0.0.1:39952)
0618 22:27:19.922203 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.922214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.922300 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.922754 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.922756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.922773 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:19.922858 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.922863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.924274 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.924277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.925512 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.925523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.926540 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.926562 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.926643 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.926665 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.926716 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.927601 (+ 885us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":11,"thread_start_us":76,"threads_started":1}
I20260618 22:27:19.930178 21840 tablet_service.cc:1467] Processing CreateTablet for tablet 8751066efd214c2fabf819b01d64a49e (DEFAULT_TABLE table=test-table [id=fb0db8a429be4620a375303b3d31bcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:19.930258 21574 tablet_service.cc:1467] Processing CreateTablet for tablet 8751066efd214c2fabf819b01d64a49e (DEFAULT_TABLE table=test-table [id=fb0db8a429be4620a375303b3d31bcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:19.930259 21707 tablet_service.cc:1467] Processing CreateTablet for tablet 8751066efd214c2fabf819b01d64a49e (DEFAULT_TABLE table=test-table [id=fb0db8a429be4620a375303b3d31bcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:19.931169 21840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8751066efd214c2fabf819b01d64a49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:19.931262 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8751066efd214c2fabf819b01d64a49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:19.931262 21574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8751066efd214c2fabf819b01d64a49e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:19.938485 21923 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Bootstrap starting.
I20260618 22:27:19.938620 21925 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Bootstrap starting.
I20260618 22:27:19.938843 21924 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Bootstrap starting.
I20260618 22:27:19.940644 21923 tablet_bootstrap.cc:654] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:19.940757 21924 tablet_bootstrap.cc:654] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:19.941148 21925 tablet_bootstrap.cc:654] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:19.941673 21923 log.cc:824] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:19.941829 21924 log.cc:824] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:19.941829 21925 log.cc:824] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:19.943749 21923 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: No bootstrap required, opened a new log
I20260618 22:27:19.943744 21925 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: No bootstrap required, opened a new log
I20260618 22:27:19.943744 21924 tablet_bootstrap.cc:492] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: No bootstrap required, opened a new log
I20260618 22:27:19.943934 21923 ts_tablet_manager.cc:1397] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:27:19.943935 21925 ts_tablet_manager.cc:1397] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:19.943935 21924 ts_tablet_manager.cc:1397] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:19.946868 21923 raft_consensus.cc:348] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.947124 21923 raft_consensus.cc:374] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:19.947203 21923 raft_consensus.cc:729] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88a7eeb369047df932acef0b2071897, State: Initialized, Role: FOLLOWER
I20260618 22:27:19.947638 21923 consensus_queue.cc:260] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.947602 21924 raft_consensus.cc:348] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.947736 21925 raft_consensus.cc:348] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.947896 21924 raft_consensus.cc:374] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:19.947962 21924 raft_consensus.cc:729] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996078b91e4a47079fa051758ec0ca7e, State: Initialized, Role: FOLLOWER
I20260618 22:27:19.948016 21925 raft_consensus.cc:374] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:19.948091 21925 raft_consensus.cc:729] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec98207ad2c42a788270d3636f13664, State: Initialized, Role: FOLLOWER
I20260618 22:27:19.948271 21906 heartbeater.cc:503] Master 127.16.243.190:43447 was elected leader, sending a full tablet report...
I20260618 22:27:19.948505 21924 consensus_queue.cc:260] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.948500 21925 consensus_queue.cc:260] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.949337 21773 heartbeater.cc:503] Master 127.16.243.190:43447 was elected leader, sending a full tablet report...
I20260618 22:27:19.949357 21640 heartbeater.cc:503] Master 127.16.243.190:43447 was elected leader, sending a full tablet report...
I20260618 22:27:19.949476 21925 ts_tablet_manager.cc:1428] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:27:19.949471 21924 ts_tablet_manager.cc:1428] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:27:19.951845 21923 ts_tablet_manager.cc:1428] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260618 22:27:19.973845 21641 tablet.cc:2369] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:19.984980 21930 raft_consensus.cc:482] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:19.985208 21930 raft_consensus.cc:504] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.986313 21930 leader_election.cc:288] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996078b91e4a47079fa051758ec0ca7e (127.16.243.129:38369), e88a7eeb369047df932acef0b2071897 (127.16.243.131:37919)
I20260618 22:27:19.990912 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.986706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:60389 (local address 127.16.243.131:37919)
0618 22:27:19.986824 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.986828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.987161 (+ 333us) server_negotiation.cc:407] Connection header received
0618 22:27:19.987239 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.987241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.987277 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.987342 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:19.987550 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.988247 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.989679 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.990302 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.990411 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.990464 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.990710 (+ 246us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.990779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:27:19.990912 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.986698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:34155 (local address 127.16.243.129:38369)
0618 22:27:19.986813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:19.986818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:19.986834 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:27:19.986941 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:19.986944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:19.986996 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:19.987064 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:19.987357 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.988136 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.989679 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.990281 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.990411 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.990477 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:19.990713 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 22:27:19.990779 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:19.990959 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.986593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:38369 (local address 127.16.243.130:34155)
0618 22:27:19.986718 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.986733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.986822 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.987103 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.987106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.987125 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:19.987257 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.987263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.988232 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.988235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.989544 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.989555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.990397 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.990401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.990523 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.990624 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.990697 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.990803 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":69,"spinlock_wait_cycles":2816}
I20260618 22:27:19.990965 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:19.986597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:37919 (local address 127.16.243.130:60389)
0618 22:27:19.987062 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:19.987074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:19.987165 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:19.987362 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:19.987365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:19.987379 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:19.987456 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.987461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.988335 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.988338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.989550 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:19.989556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:19.990405 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:19.990409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:19.990573 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:19.990624 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 22:27:19.990697 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:27:19.990803 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":246,"thread_start_us":147,"threads_started":1}
I20260618 22:27:19.991650 21594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8751066efd214c2fabf819b01d64a49e" candidate_uuid: "cec98207ad2c42a788270d3636f13664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996078b91e4a47079fa051758ec0ca7e" is_pre_election: true
I20260618 22:27:19.991650 21860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8751066efd214c2fabf819b01d64a49e" candidate_uuid: "cec98207ad2c42a788270d3636f13664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88a7eeb369047df932acef0b2071897" is_pre_election: true
I20260618 22:27:19.991972 21860 raft_consensus.cc:2454] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec98207ad2c42a788270d3636f13664 in term 0.
I20260618 22:27:19.991961 21594 raft_consensus.cc:2454] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec98207ad2c42a788270d3636f13664 in term 0.
I20260618 22:27:19.992437 21662 leader_election.cc:302] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec98207ad2c42a788270d3636f13664, e88a7eeb369047df932acef0b2071897; no voters:
I20260618 22:27:19.992693 21930 raft_consensus.cc:2781] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:19.992776 21930 raft_consensus.cc:482] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:19.992828 21930 raft_consensus.cc:3037] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:19.993830 21930 raft_consensus.cc:504] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:19.994151 21930 leader_election.cc:288] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [CANDIDATE]: Term 1 election: Requested vote from peers 996078b91e4a47079fa051758ec0ca7e (127.16.243.129:38369), e88a7eeb369047df932acef0b2071897 (127.16.243.131:37919)
I20260618 22:27:19.994534 21594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8751066efd214c2fabf819b01d64a49e" candidate_uuid: "cec98207ad2c42a788270d3636f13664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996078b91e4a47079fa051758ec0ca7e"
I20260618 22:27:19.994618 21860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8751066efd214c2fabf819b01d64a49e" candidate_uuid: "cec98207ad2c42a788270d3636f13664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88a7eeb369047df932acef0b2071897"
I20260618 22:27:19.994691 21594 raft_consensus.cc:3037] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:19.994719 21860 raft_consensus.cc:3037] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:19.995886 21860 raft_consensus.cc:2454] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec98207ad2c42a788270d3636f13664 in term 1.
I20260618 22:27:19.995886 21594 raft_consensus.cc:2454] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec98207ad2c42a788270d3636f13664 in term 1.
I20260618 22:27:19.996176 21660 leader_election.cc:302] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996078b91e4a47079fa051758ec0ca7e, cec98207ad2c42a788270d3636f13664; no voters:
I20260618 22:27:19.996371 21930 raft_consensus.cc:2781] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:19.996587 21930 raft_consensus.cc:686] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 1 LEADER]: Becoming Leader. State: Replica: cec98207ad2c42a788270d3636f13664, State: Running, Role: LEADER
I20260618 22:27:19.996881 21930 consensus_queue.cc:237] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } }
I20260618 22:27:20.000067 21450 catalog_manager.cc:5495] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec98207ad2c42a788270d3636f13664 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "cec98207ad2c42a788270d3636f13664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec98207ad2c42a788270d3636f13664" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 34417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:20.025156 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.021792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.16.243.130:34417)
0618 22:27:20.021927 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.021931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.021946 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:20.022023 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.022025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.022076 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.022162 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:20.022406 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:20.022955 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:20.023797 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:20.024013 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:20.024098 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:20.024384 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:20.024789 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:20.024893 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.024959 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.025018 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:27:20.030897 21860 raft_consensus.cc:1261] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 1 FOLLOWER]: Refusing update from remote peer cec98207ad2c42a788270d3636f13664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:20.030897 21594 raft_consensus.cc:1261] T 8751066efd214c2fabf819b01d64a49e P 996078b91e4a47079fa051758ec0ca7e [term 1 FOLLOWER]: Refusing update from remote peer cec98207ad2c42a788270d3636f13664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:20.031600 21933 consensus_queue.cc:1035] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996078b91e4a47079fa051758ec0ca7e" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:20.031771 21930 consensus_queue.cc:1035] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e88a7eeb369047df932acef0b2071897" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:20.039445 21938 mvcc.cc:205] Tried to move back new op lower bound from 7298341437559402496 to 7298341437429956608. Current Snapshot: MvccSnapshot[applied={T|T < 7298341437559402496}]
I20260618 22:27:20.047777 21450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:20.047950 21450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:20.050242 21450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9304720014cd4b74ade84c27e66e019b: Sending DeleteTablet for 3 replicas of tablet 8751066efd214c2fabf819b01d64a49e
I20260618 22:27:20.051009 21707 tablet_service.cc:1514] Processing DeleteTablet for tablet 8751066efd214c2fabf819b01d64a49e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:20 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:49104
I20260618 22:27:20.051082 21574 tablet_service.cc:1514] Processing DeleteTablet for tablet 8751066efd214c2fabf819b01d64a49e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:20 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:40474
I20260618 22:27:20.051648 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21512
I20260618 22:27:20.051738 21840 tablet_service.cc:1514] Processing DeleteTablet for tablet 8751066efd214c2fabf819b01d64a49e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:20 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:39952
I20260618 22:27:20.052281 21947 tablet_replica.cc:330] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: stopping tablet replica
I20260618 22:27:20.052634 21947 raft_consensus.cc:2229] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:20.052938 21947 raft_consensus.cc:2258] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:20.054278 21946 tablet_replica.cc:330] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: stopping tablet replica
I20260618 22:27:20.054333 21947 ts_tablet_manager.cc:1905] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:20.054601 21946 raft_consensus.cc:2229] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:20.055019 21946 raft_consensus.cc:2258] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:20.056579 21946 ts_tablet_manager.cc:1905] T 8751066efd214c2fabf819b01d64a49e P cec98207ad2c42a788270d3636f13664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:20.057130 21947 ts_tablet_manager.cc:1918] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:20.057279 21947 log.cc:1196] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/8751066efd214c2fabf819b01d64a49e
I20260618 22:27:20.057669 21947 ts_tablet_manager.cc:1939] T 8751066efd214c2fabf819b01d64a49e P e88a7eeb369047df932acef0b2071897: Deleting consensus metadata
I20260618 22:27:20.059093 21436 catalog_manager.cc:4841] TS e88a7eeb369047df932acef0b2071897 (127.16.243.131:37919): tablet 8751066efd214c2fabf819b01d64a49e (table test-table [id=fb0db8a429be4620a375303b3d31bcec]) successfully deleted
W20260618 22:27:20.059172 21434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:38369 (error 108)
I20260618 22:27:20.059391 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21645
I20260618 22:27:20.060789 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.060525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:38369 (local address 127.0.0.1:40476)
0618 22:27:20.060650 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.060716 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:38369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 22:27:20.060956 21434 catalog_manager.cc:4584] TS 996078b91e4a47079fa051758ec0ca7e (127.16.243.129:38369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8751066efd214c2fabf819b01d64a49e: Network error: Client connection negotiation failed: client connection to 127.16.243.129:38369: connect: Connection refused (error 111)
I20260618 22:27:20.064483 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21778
I20260618 22:27:20.065187 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.064665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:34417 (local address 127.0.0.1:49124)
0618 22:27:20.065044 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.065106 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
I20260618 22:27:20.068666 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21419
2026-06-18T22:27:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2143 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-18T22:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:20Z Disabled control of system clock
I20260618 22:27:20.095964 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:40363
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:40363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:27:20.204362 21952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:20.204638 21952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:20.204674 21952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:20.208668 21952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:20.208722 21952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:20.208736 21952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:20.208751 21952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:20.208767 21952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:20.213546 21952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:40363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:40363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:20.214622 21952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:20.215986 21952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:20.223984 21958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.224041 21957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.224009 21960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.224342 21952 server_base.cc:1034] running on GCE node
I20260618 22:27:20.224881 21952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:20.225991 21952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:20.227167 21952 hybrid_clock.cc:648] HybridClock initialized: now 1781821640227148 us; error 30 us; skew 500 ppm
I20260618 22:27:20.229033 21952 webserver.cc:460] Webserver started at http://127.16.243.190:33005/ using document root <none> and password file <none>
I20260618 22:27:20.229591 21952 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:20.229651 21952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:20.229864 21952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:20.231946 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "9b9de67b0ebe4aefad44d71a354a875a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "c8c164b93ab747afd161ce27859dd575"
server_key_iv: "a9fd412db3a6f3c413ac3c4a86d43ef0"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.232476 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "9b9de67b0ebe4aefad44d71a354a875a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "c8c164b93ab747afd161ce27859dd575"
server_key_iv: "a9fd412db3a6f3c413ac3c4a86d43ef0"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.236186 21952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:27:20.238682 21966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.239977 21952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:20.240114 21952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "9b9de67b0ebe4aefad44d71a354a875a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "c8c164b93ab747afd161ce27859dd575"
server_key_iv: "a9fd412db3a6f3c413ac3c4a86d43ef0"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.240227 21952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:20.245113 21952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:20.245843 21952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:20.246006 21952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:20.252377 21952 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:40363
I20260618 22:27:20.252404 22018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:40363 every 8 connection(s)
I20260618 22:27:20.253309 21952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:20.256425 22019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:20.261150 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 21952
I20260618 22:27:20.261261 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:20.261497 17358 external_mini_cluster.cc:1428] Setting key e2eb4e93109d6d85fb4be40dafb7ff5f
I20260618 22:27:20.261845 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: Bootstrap starting.
I20260618 22:27:20.264302 22019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:20.264994 22019 log.cc:824] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:20.266935 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: No bootstrap required, opened a new log
I20260618 22:27:20.267024 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.262864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.16.243.190:40363)
0618 22:27:20.263552 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.263567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.263603 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:27:20.264267 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.264284 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.264675 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.264999 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.265751 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.265756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.265768 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.265777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.265831 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.265849 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.265941 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.266066 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.266301 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":144,"threads_started":1}
I20260618 22:27:20.270709 22019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } }
I20260618 22:27:20.270936 22019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:20.270993 22019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9de67b0ebe4aefad44d71a354a875a, State: Initialized, Role: FOLLOWER
I20260618 22:27:20.271382 22019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } }
I20260618 22:27:20.271512 22019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:20.271574 22019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:20.271677 22019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:20.272645 22019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } }
I20260618 22:27:20.272933 22019 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9de67b0ebe4aefad44d71a354a875a; no voters:
I20260618 22:27:20.273177 22019 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:20.273290 22024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:20.273476 22024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [term 1 LEADER]: Becoming Leader. State: Replica: 9b9de67b0ebe4aefad44d71a354a875a, State: Running, Role: LEADER
I20260618 22:27:20.273810 22024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } }
I20260618 22:27:20.274294 22019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:20.275377 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9de67b0ebe4aefad44d71a354a875a. Latest consensus state: current_term: 1 leader_uuid: "9b9de67b0ebe4aefad44d71a354a875a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } } }
I20260618 22:27:20.275524 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:20.276033 22033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:20.275377 22025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9de67b0ebe4aefad44d71a354a875a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9de67b0ebe4aefad44d71a354a875a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40363 } } }
I20260618 22:27:20.276786 22025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:20.279256 22033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:20.284513 22033 catalog_manager.cc:1347] Generated new cluster ID: eef238283e8040fcb48580b765b6636f
I20260618 22:27:20.284647 22033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:20.308851 22033 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:20.309207 22033 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:20.321274 22033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: Generated new TSK 0
I20260618 22:27:20.322027 22033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:20.335808 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--builtin_ntp_servers=127.16.243.148:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:27:20.449683 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:20.449966 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:20.450006 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:20.454046 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:20.454118 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:20.454174 22043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:20.458968 22043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:20.460008 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:20.461311 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:20.470862 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.470839 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.470890 22043 server_base.cc:1034] running on GCE node
W20260618 22:27:20.470839 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.471613 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:20.472261 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:20.473444 22043 hybrid_clock.cc:648] HybridClock initialized: now 1781821640473413 us; error 43 us; skew 500 ppm
I20260618 22:27:20.475373 22043 webserver.cc:460] Webserver started at http://127.16.243.129:43733/ using document root <none> and password file <none>
I20260618 22:27:20.475978 22043 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:20.476048 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:20.476251 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:20.478283 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "c08bf79668384a518adba24cda356b9a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "7885f6baf29f4286a8dc6316cc1c39a0"
server_key_iv: "e97c76e4f947e893ac68738fde7d07a6"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.478843 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "c08bf79668384a518adba24cda356b9a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "7885f6baf29f4286a8dc6316cc1c39a0"
server_key_iv: "e97c76e4f947e893ac68738fde7d07a6"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.482517 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:27:20.485250 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.486621 22043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:20.486737 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "c08bf79668384a518adba24cda356b9a"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "7885f6baf29f4286a8dc6316cc1c39a0"
server_key_iv: "e97c76e4f947e893ac68738fde7d07a6"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.486848 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:20.491793 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:20.492434 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:20.492604 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:20.493149 22043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:20.494135 22043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:20.494180 22043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.494231 22043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:20.494257 22043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.503048 22043 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:36351
I20260618 22:27:20.503082 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:36351 every 8 connection(s)
I20260618 22:27:20.503970 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:20.510109 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.505847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:55569 (local address 127.16.243.190:40363)
0618 22:27:20.506014 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.506018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.507120 (+ 1102us) server_negotiation.cc:407] Connection header received
0618 22:27:20.508420 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.508423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.508468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.508541 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.509667 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.509670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.509673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.509686 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.509706 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.509714 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.509795 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.509919 (+ 124us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.509964 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:27:20.511075 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.506268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:40363 (local address 127.16.243.129:55569)
0618 22:27:20.506970 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.507012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.508213 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.508707 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.508715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.508810 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.509384 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.509386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.509429 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.509501 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.509787 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.509794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.509800 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.509959 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.510243 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260618 22:27:20.512343 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22043
I20260618 22:27:20.512430 22171 heartbeater.cc:347] Connected to a master server at 127.16.243.190:40363
I20260618 22:27:20.512462 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:20.512686 22171 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:20.512756 17358 external_mini_cluster.cc:1428] Setting key 52afdc90d8b568ac82f6493ce636138a
I20260618 22:27:20.513170 22171 heartbeater.cc:511] Master 127.16.243.190:40363 requested a full tablet report, sending...
I20260618 22:27:20.514747 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--builtin_ntp_servers=127.16.243.148:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260618 22:27:20.514810 21983 ts_manager.cc:194] Registered new tserver with Master: c08bf79668384a518adba24cda356b9a (127.16.243.129:36351)
W20260618 22:27:20.626396 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:20.626698 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:20.626739 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:20.630751 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:20.630808 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:20.630865 22174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:20.635574 22174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:20.636610 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:20.637935 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:20.646907 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.646905 22179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.647070 22174 server_base.cc:1034] running on GCE node
W20260618 22:27:20.646907 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.647599 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:20.648227 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:20.649364 22174 hybrid_clock.cc:648] HybridClock initialized: now 1781821640649332 us; error 47 us; skew 500 ppm
I20260618 22:27:20.651129 22174 webserver.cc:460] Webserver started at http://127.16.243.130:41573/ using document root <none> and password file <none>
I20260618 22:27:20.651662 22174 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:20.651721 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:20.651898 22174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:20.653891 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "58be5e91fd7240a799e142cf2d5902ba"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "39de946de1bb2c85e09c0b55eb565ea7"
server_key_iv: "7d460012c9c3b0a4d5ea6f5c4c34d0a1"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.654417 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "58be5e91fd7240a799e142cf2d5902ba"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "39de946de1bb2c85e09c0b55eb565ea7"
server_key_iv: "7d460012c9c3b0a4d5ea6f5c4c34d0a1"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.657969 22174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:20.660501 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.661744 22174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:20.661849 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "58be5e91fd7240a799e142cf2d5902ba"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "39de946de1bb2c85e09c0b55eb565ea7"
server_key_iv: "7d460012c9c3b0a4d5ea6f5c4c34d0a1"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.661942 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:20.666389 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:20.667023 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:20.667188 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:20.667706 22174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:20.668596 22174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:20.668641 22174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.668691 22174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:20.668716 22174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.676867 22174 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:33157
I20260618 22:27:20.676893 22301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:33157 every 8 connection(s)
I20260618 22:27:20.677748 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:20.683738 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.679555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:46895 (local address 127.16.243.190:40363)
0618 22:27:20.679676 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.679680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.680733 (+ 1053us) server_negotiation.cc:407] Connection header received
0618 22:27:20.682033 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.682036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.682112 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.682185 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.683289 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.683293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.683296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.683310 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.683329 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.683337 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.683419 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.683535 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.683582 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:27:20.684615 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.679927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:40363 (local address 127.16.243.130:46895)
0618 22:27:20.680590 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.680630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.681820 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.682367 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.682374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.682467 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.683029 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.683031 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.683074 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.683143 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.683405 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.683412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.683417 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.683584 (+ 167us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.683848 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260618 22:27:20.685781 22302 heartbeater.cc:347] Connected to a master server at 127.16.243.190:40363
I20260618 22:27:20.686030 22302 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:20.686443 22302 heartbeater.cc:511] Master 127.16.243.190:40363 requested a full tablet report, sending...
I20260618 22:27:20.687100 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22174
I20260618 22:27:20.687204 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:20.687507 17358 external_mini_cluster.cc:1428] Setting key 13f4be47cb9106afcab6217fc17c748d
I20260618 22:27:20.687628 21983 ts_manager.cc:194] Registered new tserver with Master: 58be5e91fd7240a799e142cf2d5902ba (127.16.243.130:33157)
I20260618 22:27:20.689335 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--builtin_ntp_servers=127.16.243.148:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:27:20.798499 22305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:20.798813 22305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:20.798874 22305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:20.802917 22305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:20.802978 22305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:20.803058 22305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:20.807987 22305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:40363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:20.809080 22305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:20.810441 22305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:20.819757 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.819770 22310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:20.819849 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:20.820021 22305 server_base.cc:1034] running on GCE node
I20260618 22:27:20.820443 22305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:20.821168 22305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:20.822362 22305 hybrid_clock.cc:648] HybridClock initialized: now 1781821640822330 us; error 42 us; skew 500 ppm
I20260618 22:27:20.824311 22305 webserver.cc:460] Webserver started at http://127.16.243.131:36883/ using document root <none> and password file <none>
I20260618 22:27:20.824936 22305 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:20.825008 22305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:20.825223 22305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:20.827539 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "81af645fd0874c95b6dd8687401391d8"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "549f8007bb4140d95f4ea50bd797cb14"
server_key_iv: "5c9f7b1a7bfd961cfad48c4916c716ba"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.828112 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "81af645fd0874c95b6dd8687401391d8"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "549f8007bb4140d95f4ea50bd797cb14"
server_key_iv: "5c9f7b1a7bfd961cfad48c4916c716ba"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.832036 22305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:20.834649 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.835875 22305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:20.835986 22305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "81af645fd0874c95b6dd8687401391d8"
format_stamp: "Formatted at 2026-06-18 22:27:20 on dist-test-slave-jm5g"
server_key: "549f8007bb4140d95f4ea50bd797cb14"
server_key_iv: "5c9f7b1a7bfd961cfad48c4916c716ba"
server_key_version: "encryptionkey@0"
I20260618 22:27:20.836097 22305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:20.840771 22305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:20.841382 22305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:20.841534 22305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:20.842059 22305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:20.842994 22305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:20.843041 22305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.843091 22305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:20.843119 22305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:20.851476 22305 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:46533
I20260618 22:27:20.851512 22432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:46533 every 8 connection(s)
I20260618 22:27:20.852394 22305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:20.855360 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22305
I20260618 22:27:20.855484 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:20.855815 17358 external_mini_cluster.cc:1428] Setting key 7eb5aa2d916b6af375648f21fdbde13e
I20260618 22:27:20.858708 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.854194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:51679 (local address 127.16.243.190:40363)
0618 22:27:20.854322 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.854326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.855567 (+ 1241us) server_negotiation.cc:407] Connection header received
0618 22:27:20.857140 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.857143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.857189 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.857267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.858339 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.858342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.858344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.858355 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.858374 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.858382 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.858448 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.858541 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.858605 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:20.859638 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.854593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:40363 (local address 127.16.243.131:51679)
0618 22:27:20.855379 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.855435 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.856926 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.857415 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.857422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.857515 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.858098 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.858099 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.858140 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.858210 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.858443 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.858447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.858450 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.858619 (+ 169us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.858844 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260618 22:27:20.860668 22433 heartbeater.cc:347] Connected to a master server at 127.16.243.190:40363
I20260618 22:27:20.860894 22433 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:20.861312 22433 heartbeater.cc:511] Master 127.16.243.190:40363 requested a full tablet report, sending...
I20260618 22:27:20.862268 21983 ts_manager.cc:194] Registered new tserver with Master: 81af645fd0874c95b6dd8687401391d8 (127.16.243.131:46533)
I20260618 22:27:20.869796 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:27:20.873123 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.871682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.16.243.190:40363)
0618 22:27:20.871814 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.871818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.871943 (+ 125us) server_negotiation.cc:407] Connection header received
0618 22:27:20.872067 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.872069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.872104 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.872161 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.872832 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.872834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.872836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.872842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.872855 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.872861 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.872915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.872964 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.872996 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:20.875945 21983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:20.878918 21983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:20.890588 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.887321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:46533 (local address 127.0.0.1:39014)
0618 22:27:20.888392 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.888444 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.888557 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.889025 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.889073 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.889152 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.889833 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.889835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.889854 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.889922 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.890229 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.890237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.890243 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.890344 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.890410 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"spinlock_wait_cycles":3328,"thread_start_us":127,"threads_started":1}
I20260618 22:27:20.890770 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.887417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.16.243.131:46533)
0618 22:27:20.888289 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.888301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.888541 (+ 240us) server_negotiation.cc:407] Connection header received
0618 22:27:20.888664 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.888669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.888880 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.889023 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.890049 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.890055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.890067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.890079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.890123 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.890157 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.890251 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.890365 (+ 114us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.890496 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":163,"threads_started":1}
I20260618 22:27:20.890961 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.887635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:33157 (local address 127.0.0.1:38768)
0618 22:27:20.888723 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.888734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.888793 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.889651 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.889654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.889666 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.890061 (+ 395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.890063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.890071 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.890091 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.890477 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.890490 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.890494 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.890818 (+ 324us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.890849 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":83,"threads_started":1}
I20260618 22:27:20.891274 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.887363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:36351 (local address 127.0.0.1:40132)
0618 22:27:20.888403 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.888444 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.888556 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.889438 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.889441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.889464 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.889839 (+ 375us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.889842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.889858 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.889924 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.890669 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.890672 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.890675 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.890743 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.890776 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"spinlock_wait_cycles":5376,"thread_start_us":93,"threads_started":1}
I20260618 22:27:20.891343 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.888163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.16.243.130:33157)
0618 22:27:20.889002 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.889014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.889036 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:27:20.889086 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.889091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.889252 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.889393 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.890209 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.890216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.890227 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.890238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.890279 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.890310 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.890409 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.890631 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.891209 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":138,"threads_started":1}
I20260618 22:27:20.891933 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.888173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.16.243.129:36351)
0618 22:27:20.888857 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.888868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.888886 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:27:20.888925 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.888929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.889107 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.889226 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.890386 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.890392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.890401 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.890452 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.890498 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.890525 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.890633 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.891721 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.891792 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":87,"threads_started":1}
I20260618 22:27:20.893816 22236 tablet_service.cc:1467] Processing CreateTablet for tablet 47ddbd780af742e1a56afee478842173 (DEFAULT_TABLE table=test-table [id=9615dd14091749bab4172159c9f38bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:20.893869 22367 tablet_service.cc:1467] Processing CreateTablet for tablet 47ddbd780af742e1a56afee478842173 (DEFAULT_TABLE table=test-table [id=9615dd14091749bab4172159c9f38bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:20.894255 22105 tablet_service.cc:1467] Processing CreateTablet for tablet 47ddbd780af742e1a56afee478842173 (DEFAULT_TABLE table=test-table [id=9615dd14091749bab4172159c9f38bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:20.894888 22367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ddbd780af742e1a56afee478842173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:20.894888 22236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ddbd780af742e1a56afee478842173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:20.895244 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ddbd780af742e1a56afee478842173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:20.900970 22450 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Bootstrap starting.
I20260618 22:27:20.902905 22451 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: Bootstrap starting.
I20260618 22:27:20.903627 22450 tablet_bootstrap.cc:654] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:20.904532 22450 log.cc:824] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:20.905121 22451 tablet_bootstrap.cc:654] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:20.905315 22452 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Bootstrap starting.
I20260618 22:27:20.905937 22451 log.cc:824] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:20.906443 22450 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: No bootstrap required, opened a new log
I20260618 22:27:20.906690 22450 ts_tablet_manager.cc:1397] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:20.907419 22452 tablet_bootstrap.cc:654] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:20.907974 22451 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: No bootstrap required, opened a new log
I20260618 22:27:20.908161 22451 ts_tablet_manager.cc:1397] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:20.908198 22452 log.cc:824] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:20.910290 22452 tablet_bootstrap.cc:492] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: No bootstrap required, opened a new log
I20260618 22:27:20.910463 22452 ts_tablet_manager.cc:1397] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:27:20.910697 22450 raft_consensus.cc:348] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.910966 22450 raft_consensus.cc:374] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:20.911021 22450 raft_consensus.cc:729] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58be5e91fd7240a799e142cf2d5902ba, State: Initialized, Role: FOLLOWER
I20260618 22:27:20.911480 22450 consensus_queue.cc:260] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.911859 22451 raft_consensus.cc:348] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.912102 22451 raft_consensus.cc:374] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:20.912149 22451 raft_consensus.cc:729] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08bf79668384a518adba24cda356b9a, State: Initialized, Role: FOLLOWER
I20260618 22:27:20.912166 22302 heartbeater.cc:503] Master 127.16.243.190:40363 was elected leader, sending a full tablet report...
I20260618 22:27:20.912496 22450 ts_tablet_manager.cc:1428] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:20.912602 22451 consensus_queue.cc:260] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.913234 22171 heartbeater.cc:503] Master 127.16.243.190:40363 was elected leader, sending a full tablet report...
I20260618 22:27:20.913553 22451 ts_tablet_manager.cc:1428] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260618 22:27:20.914450 22452 raft_consensus.cc:348] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.914705 22452 raft_consensus.cc:374] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:20.914767 22452 raft_consensus.cc:729] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81af645fd0874c95b6dd8687401391d8, State: Initialized, Role: FOLLOWER
I20260618 22:27:20.915195 22452 consensus_queue.cc:260] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.916182 22452 ts_tablet_manager.cc:1428] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:27:20.916488 22433 heartbeater.cc:503] Master 127.16.243.190:40363 was elected leader, sending a full tablet report...
W20260618 22:27:20.928669 22303 tablet.cc:2369] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:20.950771 22456 raft_consensus.cc:482] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:20.950970 22456 raft_consensus.cc:504] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.952095 22456 leader_election.cc:288] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81af645fd0874c95b6dd8687401391d8 (127.16.243.131:46533), c08bf79668384a518adba24cda356b9a (127.16.243.129:36351)
I20260618 22:27:20.953941 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.952361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:36351 (local address 127.16.243.130:39569)
0618 22:27:20.952475 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.952500 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.952585 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.952880 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.952884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.952911 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.953428 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.953430 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.953439 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.953465 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.953707 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.953709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.953712 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.953782 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.953816 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 22:27:20.953941 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.952447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39569 (local address 127.16.243.129:36351)
0618 22:27:20.952589 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.952593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.952608 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:20.952698 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.952701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.952737 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.952819 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.953594 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.953597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.953600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.953612 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.953631 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.953638 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.953715 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.953778 (+ 63us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.953816 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:20.954392 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.952452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:42103 (local address 127.16.243.131:46533)
0618 22:27:20.952579 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.952583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.953207 (+ 624us) server_negotiation.cc:407] Connection header received
0618 22:27:20.953308 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.953310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.953344 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.953407 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.954001 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.954004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.954007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.954019 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.954040 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.954055 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.954132 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.954252 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.954295 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:20.954402 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.952363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:46533 (local address 127.16.243.130:42103)
0618 22:27:20.953127 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:20.953137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:20.953224 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:20.953455 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:20.953459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:20.953474 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:20.953852 (+ 378us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:20.953855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:20.953865 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:20.953896 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:20.954169 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:20.954172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:20.954175 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:27:20.954240 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:27:20.954271 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":305,"thread_start_us":158,"threads_started":1}
I20260618 22:27:20.954605 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ddbd780af742e1a56afee478842173" candidate_uuid: "58be5e91fd7240a799e142cf2d5902ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08bf79668384a518adba24cda356b9a" is_pre_election: true
I20260618 22:27:20.954921 22125 raft_consensus.cc:2454] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58be5e91fd7240a799e142cf2d5902ba in term 0.
I20260618 22:27:20.955039 22387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ddbd780af742e1a56afee478842173" candidate_uuid: "58be5e91fd7240a799e142cf2d5902ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81af645fd0874c95b6dd8687401391d8" is_pre_election: true
I20260618 22:27:20.955303 22387 raft_consensus.cc:2454] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58be5e91fd7240a799e142cf2d5902ba in term 0.
I20260618 22:27:20.955368 22190 leader_election.cc:302] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58be5e91fd7240a799e142cf2d5902ba, c08bf79668384a518adba24cda356b9a; no voters:
I20260618 22:27:20.955653 22456 raft_consensus.cc:2781] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:20.955736 22456 raft_consensus.cc:482] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:20.955783 22456 raft_consensus.cc:3037] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:20.956800 22456 raft_consensus.cc:504] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.957131 22456 leader_election.cc:288] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [CANDIDATE]: Term 1 election: Requested vote from peers 81af645fd0874c95b6dd8687401391d8 (127.16.243.131:46533), c08bf79668384a518adba24cda356b9a (127.16.243.129:36351)
I20260618 22:27:20.957468 22387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ddbd780af742e1a56afee478842173" candidate_uuid: "58be5e91fd7240a799e142cf2d5902ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81af645fd0874c95b6dd8687401391d8"
I20260618 22:27:20.957509 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ddbd780af742e1a56afee478842173" candidate_uuid: "58be5e91fd7240a799e142cf2d5902ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08bf79668384a518adba24cda356b9a"
I20260618 22:27:20.957573 22387 raft_consensus.cc:3037] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:20.957645 22125 raft_consensus.cc:3037] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:20.958642 22387 raft_consensus.cc:2454] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58be5e91fd7240a799e142cf2d5902ba in term 1.
I20260618 22:27:20.958729 22125 raft_consensus.cc:2454] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58be5e91fd7240a799e142cf2d5902ba in term 1.
I20260618 22:27:20.958981 22189 leader_election.cc:302] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58be5e91fd7240a799e142cf2d5902ba, 81af645fd0874c95b6dd8687401391d8, c08bf79668384a518adba24cda356b9a; no voters:
I20260618 22:27:20.959167 22456 raft_consensus.cc:2781] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:20.959388 22456 raft_consensus.cc:686] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 1 LEADER]: Becoming Leader. State: Replica: 58be5e91fd7240a799e142cf2d5902ba, State: Running, Role: LEADER
I20260618 22:27:20.959745 22456 consensus_queue.cc:237] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } }
I20260618 22:27:20.962787 21982 catalog_manager.cc:5495] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 58be5e91fd7240a799e142cf2d5902ba (127.16.243.130). New cstate: current_term: 1 leader_uuid: "58be5e91fd7240a799e142cf2d5902ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58be5e91fd7240a799e142cf2d5902ba" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33157 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:20.989511 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:20.988232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.16.243.130:33157)
0618 22:27:20.988338 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:20.988341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:20.988354 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:20.988446 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:20.988448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:20.988508 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:20.988587 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:20.989175 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:20.989179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:20.989180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:20.989191 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:20.989209 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:20.989216 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:20.989299 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:20.989369 (+ 70us) server_negotiation.cc:299] Negotiation successful
0618 22:27:20.989403 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:27:20.995491 22387 raft_consensus.cc:1261] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 1 FOLLOWER]: Refusing update from remote peer 58be5e91fd7240a799e142cf2d5902ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:20.995491 22125 raft_consensus.cc:1261] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 1 FOLLOWER]: Refusing update from remote peer 58be5e91fd7240a799e142cf2d5902ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:20.996145 22456 consensus_queue.cc:1035] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08bf79668384a518adba24cda356b9a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:20.996337 22460 consensus_queue.cc:1035] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "81af645fd0874c95b6dd8687401391d8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:21.005332 22464 mvcc.cc:205] Tried to move back new op lower bound from 7298341441510551552 to 7298341441373483008. Current Snapshot: MvccSnapshot[applied={T|T < 7298341441510551552 or (T in {7298341441510551552})}]
I20260618 22:27:21.012326 21982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:21.012480 21982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:21.014917 21982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b9de67b0ebe4aefad44d71a354a875a: Sending DeleteTablet for 3 replicas of tablet 47ddbd780af742e1a56afee478842173
I20260618 22:27:21.015698 22236 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ddbd780af742e1a56afee478842173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:21 UTC) from {username='slave'} at 127.0.0.1:38768
I20260618 22:27:21.015723 22367 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ddbd780af742e1a56afee478842173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:21 UTC) from {username='slave'} at 127.0.0.1:39014
I20260618 22:27:21.015731 22105 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ddbd780af742e1a56afee478842173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:21 UTC) from {username='slave'} at 127.0.0.1:40132
I20260618 22:27:21.016250 22473 tablet_replica.cc:330] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a: stopping tablet replica
I20260618 22:27:21.016597 22473 raft_consensus.cc:2229] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:21.016933 22473 raft_consensus.cc:2258] T 47ddbd780af742e1a56afee478842173 P c08bf79668384a518adba24cda356b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:21.017519 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22043
I20260618 22:27:21.018085 22474 tablet_replica.cc:330] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: stopping tablet replica
I20260618 22:27:21.018406 22474 raft_consensus.cc:2229] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:21.018757 22474 raft_consensus.cc:2258] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:21.020900 22474 ts_tablet_manager.cc:1905] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:21.022782 22475 tablet_replica.cc:330] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: stopping tablet replica
I20260618 22:27:21.023154 22475 raft_consensus.cc:2229] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:21.023589 22475 raft_consensus.cc:2258] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:21.024089 22474 ts_tablet_manager.cc:1918] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:21.024183 22474 log.cc:1196] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/47ddbd780af742e1a56afee478842173
I20260618 22:27:21.024466 22474 ts_tablet_manager.cc:1939] T 47ddbd780af742e1a56afee478842173 P 81af645fd0874c95b6dd8687401391d8: Deleting consensus metadata
I20260618 22:27:21.025344 22475 ts_tablet_manager.cc:1905] T 47ddbd780af742e1a56afee478842173 P 58be5e91fd7240a799e142cf2d5902ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:21.025759 21967 catalog_manager.cc:4841] TS 81af645fd0874c95b6dd8687401391d8 (127.16.243.131:46533): tablet 47ddbd780af742e1a56afee478842173 (table test-table [id=9615dd14091749bab4172159c9f38bd9]) successfully deleted
W20260618 22:27:21.026260 21968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:36351 (error 108)
I20260618 22:27:21.026464 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22174
I20260618 22:27:21.027911 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.027604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:36351 (local address 127.0.0.1:40138)
0618 22:27:21.027748 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:21.027823 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:36351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260618 22:27:21.028085 21968 catalog_manager.cc:4584] TS c08bf79668384a518adba24cda356b9a (127.16.243.129:36351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47ddbd780af742e1a56afee478842173: Network error: Client connection negotiation failed: client connection to 127.16.243.129:36351: connect: Connection refused (error 111)
I20260618 22:27:21.031993 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22305
I20260618 22:27:21.033411 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.032134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:33157 (local address 127.0.0.1:38790)
0618 22:27:21.033285 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:21.033335 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1069}
I20260618 22:27:21.036446 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 21952
2026-06-18T22:27:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (966 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260618 22:27:21.055984 17358 mini_oidc.cc:152] Starting JWKS server
I20260618 22:27:21.057531 17358 webserver.cc:460] Webserver started at https://127.0.0.1:43313/ using document root <none> and password file <none>
I20260618 22:27:21.057695 17358 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:27:21.057950 17358 webserver.cc:460] Webserver started at http://127.0.0.1:40871/ using document root <none> and password file <none>
2026-06-18T22:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:21Z Disabled control of system clock
I20260618 22:27:21.066162 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43001
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43313/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/testchainca.cert with env {}
W20260618 22:27:21.179070 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:21.179371 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:21.179412 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:21.183596 22482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:21.183672 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:21.183706 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:21.183722 22482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:21.183736 22482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:27:21.183755 22482 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:27:21.183781 22482 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43313/jwks.json
I20260618 22:27:21.189493 22482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43313/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:21.190811 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:21.191972 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:21.200383 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:21.200450 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:21.200604 22482 server_base.cc:1034] running on GCE node
W20260618 22:27:21.200383 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:21.201330 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:21.202530 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:21.203727 22482 hybrid_clock.cc:648] HybridClock initialized: now 1781821641203700 us; error 43 us; skew 500 ppm
I20260618 22:27:21.205586 22482 webserver.cc:460] Webserver started at http://127.16.243.190:39995/ using document root <none> and password file <none>
I20260618 22:27:21.206148 22482 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:21.206218 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:21.206406 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:21.208477 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "56c4e5179bf64ca4b8f0371c27523ccf"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "7ed046b0892b1e1e7189c71b64fe9574"
server_key_iv: "f312e328ddebb036a467abfcd763a5d3"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.209017 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "56c4e5179bf64ca4b8f0371c27523ccf"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "7ed046b0892b1e1e7189c71b64fe9574"
server_key_iv: "f312e328ddebb036a467abfcd763a5d3"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.212828 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260618 22:27:21.215365 22496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:21.216614 22482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:21.216756 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "56c4e5179bf64ca4b8f0371c27523ccf"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "7ed046b0892b1e1e7189c71b64fe9574"
server_key_iv: "f312e328ddebb036a467abfcd763a5d3"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.216866 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:21.232205 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:21.232940 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:21.233119 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:21.240024 22482 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:43001
I20260618 22:27:21.240056 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:43001 every 8 connection(s)
I20260618 22:27:21.241009 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:21.241667 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22482
I20260618 22:27:21.241803 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:21.242105 17358 external_mini_cluster.cc:1428] Setting key 54fa6c9aa30134345ba3ed314ed4bf5e
I20260618 22:27:21.245568 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:21.251650 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.243553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.16.243.190:43001)
0618 22:27:21.244187 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.244203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.244240 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:27:21.244885 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.244905 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.245350 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.245658 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:21.246054 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.247112 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.248326 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.248812 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.248930 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.249824 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:21.249829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:21.249843 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:21.250265 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:21.250327 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:21.250347 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:21.250423 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:21.250625 (+ 202us) server_negotiation.cc:299] Negotiation successful
0618 22:27:21.250906 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260618 22:27:21.252208 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf: Bootstrap starting.
I20260618 22:27:21.254381 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:27:21.254459 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:21.255371 22549 log.cc:824] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:21.257259 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf: No bootstrap required, opened a new log
I20260618 22:27:21.260777 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } }
I20260618 22:27:21.261032 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:21.261092 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c4e5179bf64ca4b8f0371c27523ccf, State: Initialized, Role: FOLLOWER
I20260618 22:27:21.261509 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } }
I20260618 22:27:21.261638 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:21.261703 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:21.261799 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:21.262765 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } }
I20260618 22:27:21.263162 22549 leader_election.cc:302] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c4e5179bf64ca4b8f0371c27523ccf; no voters:
I20260618 22:27:21.263473 22549 leader_election.cc:288] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:21.263583 22554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:21.263816 22554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [term 1 LEADER]: Becoming Leader. State: Replica: 56c4e5179bf64ca4b8f0371c27523ccf, State: Running, Role: LEADER
I20260618 22:27:21.264128 22554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } }
I20260618 22:27:21.264659 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:21.265677 22556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c4e5179bf64ca4b8f0371c27523ccf. Latest consensus state: current_term: 1 leader_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } } }
I20260618 22:27:21.265813 22556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.266083 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c4e5179bf64ca4b8f0371c27523ccf" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43001 } } }
I20260618 22:27:21.266170 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.266251 22563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:21.269421 22563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:21.274920 22563 catalog_manager.cc:1347] Generated new cluster ID: 777db0eeb413413ba9b512ad36be52dc
I20260618 22:27:21.275009 22563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:21.289584 22563 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:21.290778 22563 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:21.296401 22563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56c4e5179bf64ca4b8f0371c27523ccf: Generated new TSK 0
I20260618 22:27:21.297175 22563 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:27:21.330968 22579 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260618 22:27:21.331096 22579 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:27:21.331614 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.307167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.16.243.190:43001)
0618 22:27:21.307309 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.307314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.307491 (+ 177us) server_negotiation.cc:407] Connection header received
0618 22:27:21.307739 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.307743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.307803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.307901 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:27:21.308200 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.308794 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.309702 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.309956 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.310057 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.310466 (+ 409us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:27:21.331155 (+ 20689us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:27:21.331472 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260618 22:27:21.331720 22552 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260618 22:27:21.332828 22578 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:21.307027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43001 (local address 127.0.0.1:48158)
0618 22:27:21.307388 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:21.307402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:21.307561 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:21.307929 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:21.307933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:21.307943 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:27:21.308082 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.308089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.308922 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.308927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.309577 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.309586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.310081 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.310085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.310180 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.310231 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:27:21.331433 (+ 21202us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:27:21.331701 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.243.190:43001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.243.190:43001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260618 22:27:21.333462 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22482
2026-06-18T22:27:21Z chronyd exiting
I20260618 22:27:21.369786 17358 test_util.cc:175] -----------------------------------------------
I20260618 22:27:21.369872 17358 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821616719733-17358-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260618 22:27:21.370898 17358 mini_oidc.cc:152] Starting JWKS server
I20260618 22:27:21.371729 17358 webserver.cc:460] Webserver started at https://127.0.0.1:46811/ using document root <none> and password file <none>
I20260618 22:27:21.371814 17358 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:27:21.372023 17358 webserver.cc:460] Webserver started at http://127.0.0.1:46101/ using document root <none> and password file <none>
2026-06-18T22:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:21Z Disabled control of system clock
I20260618 22:27:21.380261 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:39613
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:39613
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46811/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/testchainca.cert with env {}
W20260618 22:27:21.495186 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:21.495478 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:21.495528 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:21.499666 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:21.499738 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:21.499752 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:21.499768 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:21.499783 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:27:21.499799 22585 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:27:21.499810 22585 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46811/jwks.json
I20260618 22:27:21.505458 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:39613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46811/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:21.506512 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:21.507593 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:21.516378 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:21.516386 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:21.516386 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:21.516970 22585 server_base.cc:1034] running on GCE node
I20260618 22:27:21.517565 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:21.518787 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:21.519966 22585 hybrid_clock.cc:648] HybridClock initialized: now 1781821641519941 us; error 40 us; skew 500 ppm
I20260618 22:27:21.521938 22585 webserver.cc:460] Webserver started at http://127.16.243.190:37757/ using document root <none> and password file <none>
I20260618 22:27:21.522598 22585 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:21.522679 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:21.522882 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:21.525000 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "ac4744699bd4490c8f222e431e3a6d06"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "021128d2c2fa19db7837ad9dd727e3da"
server_key_iv: "679efd9301fe18d1a41037823d47a686"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.525570 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "ac4744699bd4490c8f222e431e3a6d06"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "021128d2c2fa19db7837ad9dd727e3da"
server_key_iv: "679efd9301fe18d1a41037823d47a686"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.529310 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260618 22:27:21.531886 22599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:21.533203 22585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:21.533329 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "ac4744699bd4490c8f222e431e3a6d06"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "021128d2c2fa19db7837ad9dd727e3da"
server_key_iv: "679efd9301fe18d1a41037823d47a686"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.533433 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:21.561161 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:21.561910 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:21.562094 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:21.568848 22585 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:39613
I20260618 22:27:21.568917 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:39613 every 8 connection(s)
I20260618 22:27:21.569844 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:21.573348 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:21.576367 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22585
I20260618 22:27:21.576467 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:21.576723 17358 external_mini_cluster.cc:1428] Setting key 283b02f8e8d033f1521d87b7fd0dc9f0
I20260618 22:27:21.579945 22652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06: Bootstrap starting.
I20260618 22:27:21.582943 22652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:21.583765 22652 log.cc:824] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:21.585906 22652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06: No bootstrap required, opened a new log
I20260618 22:27:21.587337 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.578141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.16.243.190:39613)
0618 22:27:21.578915 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.578938 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.578987 (+ 49us) server_negotiation.cc:407] Connection header received
0618 22:27:21.579894 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.579918 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.580470 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.580868 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:21.581365 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.582871 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.583830 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.584463 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.584632 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.585591 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:21.585598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:21.585611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:21.586012 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:21.586087 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:21.586109 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:21.586193 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:21.586331 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 22:27:21.586636 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":111,"threads_started":1}
I20260618 22:27:21.589805 22652 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } }
I20260618 22:27:21.590095 22652 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:21.590155 22652 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4744699bd4490c8f222e431e3a6d06, State: Initialized, Role: FOLLOWER
I20260618 22:27:21.590343 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:27:21.590634 22652 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } }
I20260618 22:27:21.590766 22652 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:21.590826 22652 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:21.590917 22652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:21.591985 22652 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } }
I20260618 22:27:21.592331 22652 leader_election.cc:302] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4744699bd4490c8f222e431e3a6d06; no voters:
I20260618 22:27:21.592628 22652 leader_election.cc:288] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:21.592752 22657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:21.592954 22657 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [term 1 LEADER]: Becoming Leader. State: Replica: ac4744699bd4490c8f222e431e3a6d06, State: Running, Role: LEADER
I20260618 22:27:21.593523 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } }
I20260618 22:27:21.593865 22652 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:21.595117 22659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4744699bd4490c8f222e431e3a6d06. Latest consensus state: current_term: 1 leader_uuid: "ac4744699bd4490c8f222e431e3a6d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } } }
I20260618 22:27:21.595149 22658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4744699bd4490c8f222e431e3a6d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4744699bd4490c8f222e431e3a6d06" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39613 } } }
I20260618 22:27:21.595237 22659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.595239 22658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.595574 22667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:21.598194 22667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:21.603425 22667 catalog_manager.cc:1347] Generated new cluster ID: 0155f1b0b06b432280ca69bc936e471f
I20260618 22:27:21.603504 22667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:21.625818 22667 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:21.627213 22667 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:21.637909 22667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4744699bd4490c8f222e431e3a6d06: Generated new TSK 0
I20260618 22:27:21.638811 22667 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:27:21.667100 22682 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260618 22:27:21.667182 22682 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:27:21.667965 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.651333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.16.243.190:39613)
0618 22:27:21.651494 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.651499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.651605 (+ 106us) server_negotiation.cc:407] Connection header received
0618 22:27:21.651780 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.651783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.651846 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.651948 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:27:21.652143 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.652981 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.653853 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.654196 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.654289 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.654596 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:27:21.667530 (+ 12934us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:27:21.667821 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260618 22:27:21.668041 22655 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260618 22:27:21.668057 22681 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:21.651218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:39613 (local address 127.0.0.1:46390)
0618 22:27:21.651518 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:21.651530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:21.651633 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:21.651949 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:21.651951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:21.651957 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:27:21.652049 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.652053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.653091 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.653093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.653742 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.653751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.654307 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.654310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.654369 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.654393 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:27:21.667751 (+ 13358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:27:21.667908 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.243.190:39613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46811/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260618 22:27:21.668747 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22585
2026-06-18T22:27:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260618 22:27:21.706660 17358 mini_oidc.cc:152] Starting JWKS server
I20260618 22:27:21.707505 17358 webserver.cc:460] Webserver started at https://127.0.0.1:37457/ using document root <none> and password file <none>
I20260618 22:27:21.707592 17358 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:27:21.707847 17358 webserver.cc:460] Webserver started at http://127.0.0.1:37563/ using document root <none> and password file <none>
2026-06-18T22:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:21Z Disabled control of system clock
I20260618 22:27:21.716046 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:40809
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:41679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:40809
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:27:21.824884 22688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:21.825191 22688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:21.825248 22688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:21.829442 22688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:21.829526 22688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:21.829548 22688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:21.829569 22688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:21.829591 22688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:27:21.829628 22688 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:27:21.829656 22688 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37457/jwks.json
I20260618 22:27:21.834789 22688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:40809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:21.835887 22688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:21.837249 22688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:21.845752 22694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:21.845769 22696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:21.845824 22688 server_base.cc:1034] running on GCE node
W20260618 22:27:21.845764 22693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:21.846704 22688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:21.847791 22688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:21.848958 22688 hybrid_clock.cc:648] HybridClock initialized: now 1781821641848944 us; error 28 us; skew 500 ppm
I20260618 22:27:21.851008 22688 webserver.cc:460] Webserver started at http://127.16.243.190:44913/ using document root <none> and password file <none>
I20260618 22:27:21.851578 22688 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:21.851639 22688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:21.851824 22688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:21.853894 22688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "62fda5f4d3224866b46bd8deb4fa925a"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "c63a13f8cb45c984dd4a3016db0ea7f0"
server_key_iv: "77dbcafc218bf54da66d4f40546d65f4"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.854418 22688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "62fda5f4d3224866b46bd8deb4fa925a"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "c63a13f8cb45c984dd4a3016db0ea7f0"
server_key_iv: "77dbcafc218bf54da66d4f40546d65f4"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.858242 22688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:27:21.860891 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:21.862213 22688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:21.862339 22688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "62fda5f4d3224866b46bd8deb4fa925a"
format_stamp: "Formatted at 2026-06-18 22:27:21 on dist-test-slave-jm5g"
server_key: "c63a13f8cb45c984dd4a3016db0ea7f0"
server_key_iv: "77dbcafc218bf54da66d4f40546d65f4"
server_key_version: "encryptionkey@0"
I20260618 22:27:21.862450 22688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:21.872439 22688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:21.873119 22688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:21.873291 22688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:21.880002 22688 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:40809
I20260618 22:27:21.880064 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:40809 every 8 connection(s)
I20260618 22:27:21.880997 22688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:21.881346 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22688
I20260618 22:27:21.881449 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:21.881732 17358 external_mini_cluster.cc:1428] Setting key ec1039d2e16fe3aef7601a3cf1248dda
I20260618 22:27:21.884559 22755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:21.891512 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a: Bootstrap starting.
I20260618 22:27:21.892376 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.883134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.16.243.190:40809)
0618 22:27:21.883819 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.883840 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.883884 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:27:21.884726 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.884749 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.885241 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.885615 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:21.886044 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.887553 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.888522 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.889212 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.889372 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.890203 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:21.890210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:21.890229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:21.890800 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:21.890873 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:21.890899 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:21.890992 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:21.891145 (+ 153us) server_negotiation.cc:299] Negotiation successful
0618 22:27:21.891426 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260618 22:27:21.894172 22755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:21.895008 22755 log.cc:824] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:21.895507 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:27:21.897030 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a: No bootstrap required, opened a new log
I20260618 22:27:21.900452 22755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } }
I20260618 22:27:21.900692 22755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:21.900751 22755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fda5f4d3224866b46bd8deb4fa925a, State: Initialized, Role: FOLLOWER
I20260618 22:27:21.901181 22755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } }
I20260618 22:27:21.901304 22755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:21.901348 22755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:21.901417 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:21.902275 22755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } }
I20260618 22:27:21.902597 22755 leader_election.cc:302] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fda5f4d3224866b46bd8deb4fa925a; no voters:
I20260618 22:27:21.902865 22755 leader_election.cc:288] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:21.902999 22760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:21.903219 22760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [term 1 LEADER]: Becoming Leader. State: Replica: 62fda5f4d3224866b46bd8deb4fa925a, State: Running, Role: LEADER
I20260618 22:27:21.903491 22760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } }
I20260618 22:27:21.904032 22755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:21.905061 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62fda5f4d3224866b46bd8deb4fa925a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } } }
I20260618 22:27:21.905206 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.905452 22762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fda5f4d3224866b46bd8deb4fa925a. Latest consensus state: current_term: 1 leader_uuid: "62fda5f4d3224866b46bd8deb4fa925a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fda5f4d3224866b46bd8deb4fa925a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 40809 } } }
I20260618 22:27:21.905552 22762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:21.905681 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:21.908838 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:21.914134 22769 catalog_manager.cc:1347] Generated new cluster ID: 724278278bb641cfa8d3215c1b73d698
I20260618 22:27:21.914213 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:21.925874 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:21.927199 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:21.939481 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62fda5f4d3224866b46bd8deb4fa925a: Generated new TSK 0
I20260618 22:27:21.940377 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:21.953809 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:21.948692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.16.243.190:40809)
0618 22:27:21.948815 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:21.948820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:21.948949 (+ 129us) server_negotiation.cc:407] Connection header received
0618 22:27:21.949093 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:21.949097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:21.949154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:21.949293 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:27:21.949465 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.950290 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.951189 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.951540 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.951642 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.951897 (+ 255us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:27:21.953420 (+ 1523us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:27:21.953664 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45548: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260618 22:27:21.953886 22758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45548: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260618 22:27:21.955421 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:21.948588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:40809 (local address 127.0.0.1:45548)
0618 22:27:21.948854 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:21.948866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:21.948963 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:21.949280 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:21.949282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:21.949287 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:27:21.949374 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.949378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.950406 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.950408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.951075 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:21.951086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:21.951652 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:21.951655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:21.951713 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:21.951735 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:27:21.953605 (+ 1870us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:27:21.955246 (+ 1641us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.243.190:40809: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1448,"thread_start_us":95,"threads_started":1}
I20260618 22:27:21.956025 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22688
E20260618 22:27:21.964272 22785 webserver.cc:533] Webserver: sslize failed:
E20260618 22:27:21.964354 22785 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-18T22:27:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-18T22:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:21Z Disabled control of system clock
I20260618 22:27:22.002398 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34689
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:34689
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/testchainca.cert with env {}
W20260618 22:27:22.112571 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:22.112865 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:22.112906 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:22.117036 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:22.117113 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:22.117127 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:22.117143 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:22.117156 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:27:22.117172 22789 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:27:22.117183 22789 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260618 22:27:22.122788 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:34689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:22.123886 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:22.124864 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:22.133235 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:22.133319 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:22.133255 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:22.133487 22789 server_base.cc:1034] running on GCE node
I20260618 22:27:22.134168 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:22.135375 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:22.136552 22789 hybrid_clock.cc:648] HybridClock initialized: now 1781821642136527 us; error 40 us; skew 500 ppm
I20260618 22:27:22.138522 22789 webserver.cc:460] Webserver started at http://127.16.243.190:39319/ using document root <none> and password file <none>
I20260618 22:27:22.139145 22789 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:22.139218 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:22.139410 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:22.141543 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "5f658051405d452f982f29247f09c874"
format_stamp: "Formatted at 2026-06-18 22:27:22 on dist-test-slave-jm5g"
server_key: "5f588467cfa70800b843607bf6f0581b"
server_key_iv: "f5a62c7e4f2d014b03b3eec3beb69089"
server_key_version: "encryptionkey@0"
I20260618 22:27:22.142117 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "5f658051405d452f982f29247f09c874"
format_stamp: "Formatted at 2026-06-18 22:27:22 on dist-test-slave-jm5g"
server_key: "5f588467cfa70800b843607bf6f0581b"
server_key_iv: "f5a62c7e4f2d014b03b3eec3beb69089"
server_key_version: "encryptionkey@0"
I20260618 22:27:22.146116 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:22.148731 22803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:22.150121 22789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:22.150262 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "5f658051405d452f982f29247f09c874"
format_stamp: "Formatted at 2026-06-18 22:27:22 on dist-test-slave-jm5g"
server_key: "5f588467cfa70800b843607bf6f0581b"
server_key_iv: "f5a62c7e4f2d014b03b3eec3beb69089"
server_key_version: "encryptionkey@0"
I20260618 22:27:22.150393 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:22.169874 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:22.170639 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:22.170809 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:22.177467 22789 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:34689
I20260618 22:27:22.177484 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:34689 every 8 connection(s)
I20260618 22:27:22.178431 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:22.181249 22856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:22.186892 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874: Bootstrap starting.
I20260618 22:27:22.188277 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22789
I20260618 22:27:22.188390 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:22.188630 17358 external_mini_cluster.cc:1428] Setting key 7572ae4de58d222a92694a51dcda7231
I20260618 22:27:22.189525 22856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:22.190246 22856 log.cc:824] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:22.192278 22856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874: No bootstrap required, opened a new log
I20260618 22:27:22.196005 22856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } }
I20260618 22:27:22.196247 22856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:22.196300 22856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f658051405d452f982f29247f09c874, State: Initialized, Role: FOLLOWER
I20260618 22:27:22.196758 22856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } }
I20260618 22:27:22.196892 22856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:22.196959 22856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:22.197057 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:22.197464 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:22.189692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.16.243.190:34689)
0618 22:27:22.190335 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:22.190352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:22.190396 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:27:22.191096 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:22.191114 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:22.191459 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:22.191749 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:22.192088 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.193106 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.194072 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.194608 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.194759 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:22.195617 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:22.195622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:22.195636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:22.196047 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:22.196106 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:22.196125 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:22.196207 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:22.196347 (+ 140us) server_negotiation.cc:299] Negotiation successful
0618 22:27:22.196629 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260618 22:27:22.198171 22856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } }
I20260618 22:27:22.198535 22856 leader_election.cc:302] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f658051405d452f982f29247f09c874; no voters:
I20260618 22:27:22.198871 22856 leader_election.cc:288] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:22.198962 22861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:22.199134 22861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [term 1 LEADER]: Becoming Leader. State: Replica: 5f658051405d452f982f29247f09c874, State: Running, Role: LEADER
I20260618 22:27:22.199431 22861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } }
I20260618 22:27:22.200206 22856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:22.200598 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:27:22.201081 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f658051405d452f982f29247f09c874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } } }
I20260618 22:27:22.201197 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:22.201450 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f658051405d452f982f29247f09c874. Latest consensus state: current_term: 1 leader_uuid: "5f658051405d452f982f29247f09c874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f658051405d452f982f29247f09c874" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34689 } } }
I20260618 22:27:22.201542 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:22.201815 22870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:22.204910 22870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:22.211161 22870 catalog_manager.cc:1347] Generated new cluster ID: d8e22960dfdc432683426fc82189832f
I20260618 22:27:22.211236 22870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:22.227484 22870 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:22.228564 22870 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:22.236418 22870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f658051405d452f982f29247f09c874: Generated new TSK 0
I20260618 22:27:22.237294 22870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:22.354663 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:22.249568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.16.243.190:34689)
0618 22:27:22.249685 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:22.249690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:22.249880 (+ 190us) server_negotiation.cc:407] Connection header received
0618 22:27:22.250063 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:22.250065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:22.250114 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:22.250209 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:27:22.250446 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.251051 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.251919 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.252148 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.252236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:22.252563 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:27:22.354142 (+101579us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:27:22.354480 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260618 22:27:22.354756 22859 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260618 22:27:22.354816 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:22.249500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34689 (local address 127.0.0.1:51972)
0618 22:27:22.249795 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:22.249807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:22.249921 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:22.250221 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:22.250224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:22.250233 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:27:22.250344 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:22.250350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.251165 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.251169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:22.251803 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:22.251812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:22.252268 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:22.252273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:22.252369 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:22.252392 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:27:22.354403 (+102011us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:27:22.354636 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.243.190:34689: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260618 22:27:22.355473 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22789
2026-06-18T22:27:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (379 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:22 dist-test-slave-jm5g krb5kdc[22897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:22 dist-test-slave-jm5g krb5kdc[22897](info): set up 2 sockets
Jun 18 22:27:22 dist-test-slave-jm5g krb5kdc[22897](info): commencing operation
krb5kdc: starting...
W20260618 22:27:23.546594 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:23 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:23.659166 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:36963
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:23.766100 22910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:23.766402 22910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:23.766460 22910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:23.770538 22910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:23.770638 22910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:23.770664 22910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:23.770687 22910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:23.770705 22910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:23.776369 22910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:23.777455 22910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:23.778478 22910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:23.786832 22915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:23.786861 22916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:23.786885 22918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:23.786841 22910 server_base.cc:1034] running on GCE node
I20260618 22:27:23.787648 22910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:23.788805 22910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:23.789970 22910 hybrid_clock.cc:648] HybridClock initialized: now 1781821643789953 us; error 34 us; skew 500 ppm
Jun 18 22:27:23 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821643, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:23.792752 22910 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:23.793665 22910 webserver.cc:460] Webserver started at http://127.16.243.190:45129/ using document root <none> and password file <none>
I20260618 22:27:23.794234 22910 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:23.794305 22910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:23.794502 22910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:23.796641 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "d3c77e689dad484881c960d38bd721e8"
format_stamp: "Formatted at 2026-06-18 22:27:23 on dist-test-slave-jm5g"
server_key: "58659d9f4df910724c0664aba882509e"
server_key_iv: "680b76788335b836ac3c6941d1e571e5"
server_key_version: "encryptionkey@0"
I20260618 22:27:23.797180 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "d3c77e689dad484881c960d38bd721e8"
format_stamp: "Formatted at 2026-06-18 22:27:23 on dist-test-slave-jm5g"
server_key: "58659d9f4df910724c0664aba882509e"
server_key_iv: "680b76788335b836ac3c6941d1e571e5"
server_key_version: "encryptionkey@0"
I20260618 22:27:23.800988 22910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260618 22:27:23.803562 22925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:23.804970 22910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:23.805115 22910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "d3c77e689dad484881c960d38bd721e8"
format_stamp: "Formatted at 2026-06-18 22:27:23 on dist-test-slave-jm5g"
server_key: "58659d9f4df910724c0664aba882509e"
server_key_iv: "680b76788335b836ac3c6941d1e571e5"
server_key_version: "encryptionkey@0"
I20260618 22:27:23.805238 22910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:23.828994 22910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:23.832700 22910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:23.832940 22910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:23.840610 22910 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:36963
I20260618 22:27:23.840623 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:36963 every 8 connection(s)
I20260618 22:27:23.841665 22910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:23.844583 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:23.845211 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 22910
I20260618 22:27:23.845319 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:23.845582 17358 external_mini_cluster.cc:1428] Setting key 724fb7b567d33a58662c4e8182a87ab4
I20260618 22:27:23.851398 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8: Bootstrap starting.
I20260618 22:27:23.853945 22978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:23.854739 22978 log.cc:824] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:27:23 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:23.856879 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8: No bootstrap required, opened a new log
I20260618 22:27:23.860267 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:23.847019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.16.243.190:36963)
0618 22:27:23.847469 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:23.847489 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:23.847536 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:27:23.848360 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:23.848384 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:23.848817 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:23.849212 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:23.849933 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:23.851414 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:23.852376 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:23.853089 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:23.853240 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:23.855796 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:23.855801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:23.855816 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:23.855842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:23.857472 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:23.857898 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:23.857903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:23.857908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:23.858018 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:23.858279 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:23.858283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:23.858284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:23.858717 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:23.858901 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:23.859174 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 22:27:23.859460 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260618 22:27:23.860442 22978 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } }
I20260618 22:27:23.860703 22978 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:23.860760 22978 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c77e689dad484881c960d38bd721e8, State: Initialized, Role: FOLLOWER
I20260618 22:27:23.861204 22978 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } }
I20260618 22:27:23.861331 22978 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:23.861385 22978 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:23.861482 22978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:23.862449 22978 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } }
I20260618 22:27:23.862812 22978 leader_election.cc:302] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c77e689dad484881c960d38bd721e8; no voters:
I20260618 22:27:23.863093 22978 leader_election.cc:288] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:23.863238 22983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:23.863444 22983 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [term 1 LEADER]: Becoming Leader. State: Replica: d3c77e689dad484881c960d38bd721e8, State: Running, Role: LEADER
I20260618 22:27:23.863754 22983 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } }
I20260618 22:27:23.864293 22978 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:23.865576 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c77e689dad484881c960d38bd721e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } } }
I20260618 22:27:23.865700 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:23.866026 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c77e689dad484881c960d38bd721e8. Latest consensus state: current_term: 1 leader_uuid: "d3c77e689dad484881c960d38bd721e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c77e689dad484881c960d38bd721e8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 36963 } } }
I20260618 22:27:23.866127 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:23.866065 22993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:23.868875 22993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:23.874641 22993 catalog_manager.cc:1347] Generated new cluster ID: 61db23d28e4041ef894942fab8d8db30
I20260618 22:27:23.874722 22993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:23.893813 22993 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:23.894881 22993 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:23.912007 22993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c77e689dad484881c960d38bd721e8: Generated new TSK 0
I20260618 22:27:23.912732 22993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:23.941457 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:24.050748 23004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:24.051025 23004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:24.051086 23004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:24.055239 23004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:24.055313 23004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:24.055397 23004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:24.060636 23004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:24.061700 23004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:24.062779 23004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:24.072439 23004 server_base.cc:1034] running on GCE node
W20260618 22:27:24.072351 23009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:24.072347 23010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:24.072351 23012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:24.072992 23004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:24.073597 23004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:24.074749 23004 hybrid_clock.cc:648] HybridClock initialized: now 1781821644074730 us; error 37 us; skew 500 ppm
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.077448 23004 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:24.078348 23004 webserver.cc:460] Webserver started at http://127.16.243.129:46573/ using document root <none> and password file <none>
I20260618 22:27:24.078953 23004 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:24.079020 23004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:24.079222 23004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:24.081252 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "a6dbca7b050c4f63bf94cfb9d19fc8ad"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "073fad2b0604d543366c7adb34e8a62d"
server_key_iv: "8e7530efb83d73d845a1e1f3d2da0e75"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.081800 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "a6dbca7b050c4f63bf94cfb9d19fc8ad"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "073fad2b0604d543366c7adb34e8a62d"
server_key_iv: "8e7530efb83d73d845a1e1f3d2da0e75"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.085527 23004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:24.088001 23019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.089222 23004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:24.089341 23004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "a6dbca7b050c4f63bf94cfb9d19fc8ad"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "073fad2b0604d543366c7adb34e8a62d"
server_key_iv: "8e7530efb83d73d845a1e1f3d2da0e75"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.089440 23004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:24.109177 23004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:24.112397 23004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:24.112581 23004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:24.113169 23004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:24.113992 23004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:24.114037 23004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.114089 23004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:24.114116 23004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.122288 23004 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:35213
I20260618 22:27:24.122319 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:35213 every 8 connection(s)
I20260618 22:27:24.123204 23004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:24.127815 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23004
I20260618 22:27:24.127920 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:24.128162 17358 external_mini_cluster.cc:1428] Setting key 2d1587012c2eff691c4650f11ec28c07
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:24.137226 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.124897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:36569 (local address 127.16.243.190:36963)
0618 22:27:24.125055 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.125060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.125899 (+ 839us) server_negotiation.cc:407] Connection header received
0618 22:27:24.127229 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.127232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.127279 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.127353 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.128604 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.129136 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.130332 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.130670 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.130771 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.133600 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.133605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.133608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.133635 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.135178 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.135680 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.135684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.135686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.135763 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.136086 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.136089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.136091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.136274 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.136355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.136900 (+ 545us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.137044 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:27:24.138125 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.125310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36963 (local address 127.16.243.129:36569)
0618 22:27:24.125751 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:24.125796 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:24.127041 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:24.127522 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:24.127530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:24.127904 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:24.128436 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.128450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.129285 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.129289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.130215 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.130223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.130822 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.130826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.131030 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.131701 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:24.131724 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:24.133429 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:24.135292 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.135298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.135316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.135585 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.135845 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.135848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.135850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.135994 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.136369 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:24.136376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:24.136652 (+ 276us) client_negotiation.cc:769] Sending connection context
0618 22:27:24.136849 (+ 197us) client_negotiation.cc:240] Negotiation successful
0618 22:27:24.137191 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:24.139472 23133 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36963
I20260618 22:27:24.139721 23133 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:24.140228 23133 heartbeater.cc:511] Master 127.16.243.190:36963 requested a full tablet report, sending...
I20260618 22:27:24.141727 22942 ts_manager.cc:194] Registered new tserver with Master: a6dbca7b050c4f63bf94cfb9d19fc8ad (127.16.243.129:35213)
I20260618 22:27:24.142884 22942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:36569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:24.153432 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:24.262583 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:24.262857 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:24.262900 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:24.267133 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:24.267205 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:24.267270 23138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:24.272621 23138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:24.273664 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:24.274708 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:24.284099 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:24.284109 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:24.284231 23138 server_base.cc:1034] running on GCE node
W20260618 22:27:24.284101 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:24.284837 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:24.285497 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:24.286604 23138 hybrid_clock.cc:648] HybridClock initialized: now 1781821644286572 us; error 42 us; skew 500 ppm
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.294512 23138 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:24.295490 23138 webserver.cc:460] Webserver started at http://127.16.243.130:33053/ using document root <none> and password file <none>
I20260618 22:27:24.296044 23138 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:24.296106 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:24.296309 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:24.298466 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "ee745ee671bf4ccf91a6ed2d45f9d68c"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "c10b88cd234d7200e9ea4e5339dd08ca"
server_key_iv: "bf62e365bbb71ed544655d04a2925b77"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.299088 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "ee745ee671bf4ccf91a6ed2d45f9d68c"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "c10b88cd234d7200e9ea4e5339dd08ca"
server_key_iv: "bf62e365bbb71ed544655d04a2925b77"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.302875 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:27:24.305377 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.306659 23138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:24.306764 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "ee745ee671bf4ccf91a6ed2d45f9d68c"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "c10b88cd234d7200e9ea4e5339dd08ca"
server_key_iv: "bf62e365bbb71ed544655d04a2925b77"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.306869 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:24.334991 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:24.337940 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:24.338109 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:24.338819 23138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:24.339723 23138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:24.339773 23138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.339814 23138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:24.339840 23138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.348232 23138 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36657
I20260618 22:27:24.348254 23266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36657 every 8 connection(s)
I20260618 22:27:24.349152 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:24.349634 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23138
I20260618 22:27:24.349753 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:24.350026 17358 external_mini_cluster.cc:1428] Setting key eb21a2e70967582ac3c0647913f722e0
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:24.366832 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.350944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55325 (local address 127.16.243.190:36963)
0618 22:27:24.351080 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.351092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.352157 (+ 1065us) server_negotiation.cc:407] Connection header received
0618 22:27:24.353754 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.353758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.353817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.353911 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.355450 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.356039 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.357761 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.357981 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.358063 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.363030 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.363034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.363037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.363069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.364461 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.365072 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.365076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.365078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.365178 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.365491 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.365494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.365496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.365735 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.365825 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.366464 (+ 639us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.366623 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:24.367654 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.351448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36963 (local address 127.16.243.130:55325)
0618 22:27:24.351988 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:24.352045 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:24.353524 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:24.354107 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:24.354115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:24.354616 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:24.355283 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.355304 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.356200 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.356204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.357618 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.357630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.358139 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.358144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.358350 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.359004 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:24.359026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:24.362833 (+ 3807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:24.364611 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.364618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.364636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.364943 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.365262 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.365264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.365266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.365408 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.365836 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:24.365842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:24.366157 (+ 315us) client_negotiation.cc:769] Sending connection context
0618 22:27:24.366374 (+ 217us) client_negotiation.cc:240] Negotiation successful
0618 22:27:24.366753 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260618 22:27:24.368765 23267 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36963
I20260618 22:27:24.369035 23267 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:24.369676 23267 heartbeater.cc:511] Master 127.16.243.190:36963 requested a full tablet report, sending...
I20260618 22:27:24.370817 22942 ts_manager.cc:194] Registered new tserver with Master: ee745ee671bf4ccf91a6ed2d45f9d68c (127.16.243.130:36657)
I20260618 22:27:24.371462 22942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:55325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:24.377625 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:24.486698 23272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:24.486970 23272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:24.487015 23272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:24.491029 23272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:24.491093 23272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:24.491159 23272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:24.496527 23272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:36963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:24.497617 23272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:24.498646 23272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:24.508131 23280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:24.508131 23278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:24.508203 23272 server_base.cc:1034] running on GCE node
W20260618 22:27:24.508179 23277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:24.508846 23272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:24.509532 23272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:24.510695 23272 hybrid_clock.cc:648] HybridClock initialized: now 1781821644510667 us; error 45 us; skew 500 ppm
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.513517 23272 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:24.514427 23272 webserver.cc:460] Webserver started at http://127.16.243.131:42607/ using document root <none> and password file <none>
I20260618 22:27:24.515026 23272 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:24.515096 23272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:24.515300 23272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:24.517330 23272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "ab714273d6894bd0a1e2135ebbb5bf49"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "59330f814bfb54a96450d13a8afa1227"
server_key_iv: "efb7ff7bfd317378ceaf05841e60b4e7"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.517858 23272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "ab714273d6894bd0a1e2135ebbb5bf49"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "59330f814bfb54a96450d13a8afa1227"
server_key_iv: "efb7ff7bfd317378ceaf05841e60b4e7"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.521606 23272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:24.524169 23287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.525444 23272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:24.525552 23272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "ab714273d6894bd0a1e2135ebbb5bf49"
format_stamp: "Formatted at 2026-06-18 22:27:24 on dist-test-slave-jm5g"
server_key: "59330f814bfb54a96450d13a8afa1227"
server_key_iv: "efb7ff7bfd317378ceaf05841e60b4e7"
server_key_version: "encryptionkey@0"
I20260618 22:27:24.525669 23272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:24.543799 23272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:24.546939 23272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:24.547117 23272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:24.547675 23272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:24.548460 23272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:24.548506 23272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.548548 23272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:24.548576 23272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:24.556547 23272 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:33957
I20260618 22:27:24.556591 23400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:33957 every 8 connection(s)
I20260618 22:27:24.557412 23272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:24.563820 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23272
I20260618 22:27:24.563934 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:24.564220 17358 external_mini_cluster.cc:1428] Setting key 731925ab61d17e834e7afb10a0d0380d
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:24.570602 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.559167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:46281 (local address 127.16.243.190:36963)
0618 22:27:24.559298 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.559302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.560103 (+ 801us) server_negotiation.cc:407] Connection header received
0618 22:27:24.561410 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.561414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.561468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.561539 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.562717 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.563241 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.564506 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.564791 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.564882 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.567534 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.567538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.567541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.567567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.568661 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.569126 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.569129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.569131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.569220 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.569522 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.569525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.569527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.569714 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.569797 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.570292 (+ 495us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.570412 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:27:24.571403 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.559550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:36963 (local address 127.16.243.131:46281)
0618 22:27:24.559966 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:24.560007 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:24.561208 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:24.561715 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:24.561722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:24.562069 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:24.562581 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.562601 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.563359 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.563362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.564389 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:24.564399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.564930 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.564935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:24.565164 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.565772 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:24.565792 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:24.567382 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:24.568763 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.568769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.568786 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.569036 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.569300 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:24.569303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:24.569305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:24.569438 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:24.569808 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:24.569814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:24.570061 (+ 247us) client_negotiation.cc:769] Sending connection context
0618 22:27:24.570237 (+ 176us) client_negotiation.cc:240] Negotiation successful
0618 22:27:24.570591 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260618 22:27:24.572459 23401 heartbeater.cc:347] Connected to a master server at 127.16.243.190:36963
I20260618 22:27:24.572677 23401 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:24.573175 23401 heartbeater.cc:511] Master 127.16.243.190:36963 requested a full tablet report, sending...
I20260618 22:27:24.574156 22942 ts_manager.cc:194] Registered new tserver with Master: ab714273d6894bd0a1e2135ebbb5bf49 (127.16.243.131:33957)
I20260618 22:27:24.574751 22942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:46281
I20260618 22:27:24.578267 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/test-admin.keytab.
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.592792 17358 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260618 22:27:24.595583 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:24.594200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.16.243.129:35213)
0618 22:27:24.594535 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.594590 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.594616 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:27:24.594667 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.594672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.594856 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.594971 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.595403 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56274: BlockingRecv error: recv got EOF from 127.0.0.1:56274 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
W20260618 22:27:24.595901 17358 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc)
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.596864 17358 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.597772 17358 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:24.606990 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.598708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.16.243.129:35213)
0618 22:27:24.598839 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.598844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.598909 (+ 65us) server_negotiation.cc:407] Connection header received
0618 22:27:24.599008 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.599010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.599050 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.599116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.599577 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.600425 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.601272 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.601633 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.601708 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.603919 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.603928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.603942 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.603965 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.605462 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.605844 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.605849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.605850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.605924 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.606122 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.606125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.606126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.606355 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.606495 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.606689 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.606824 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260618 22:27:24.610713 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:24.609732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.16.243.129:35213)
0618 22:27:24.609879 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.609883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.609916 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:27:24.610057 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.610059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.610093 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.610152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.610613 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56300: BlockingRecv error: recv got EOF from 127.0.0.1:56300 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260618 22:27:24.610800 17358 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.611773 17358 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.612612 17358 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:24.620445 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.613395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.16.243.129:35213)
0618 22:27:24.613515 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.613519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.613567 (+ 48us) server_negotiation.cc:407] Connection header received
0618 22:27:24.613667 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.613669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.613702 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.613761 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.614179 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.614779 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.615541 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.615770 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.615847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.617769 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.617772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.617774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.617796 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.619056 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.619426 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.619428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.619429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.619512 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.619720 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.619722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.619723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.619870 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.619958 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.620163 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.620286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 22:27:24.622990 23408 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:24.622094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.16.243.129:35213)
0618 22:27:24.622218 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.622222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.622299 (+ 77us) server_negotiation.cc:407] Connection header received
0618 22:27:24.622406 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.622408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.622448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.622510 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.622893 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56316: BlockingRecv error: recv got EOF from 127.0.0.1:56316 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260618 22:27:24.623078 17358 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821616719733-17358-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.624004 17358 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:24.624785 17358 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[22897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:27:24.632472 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:24.625521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.16.243.129:35213)
0618 22:27:24.625648 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:24.625651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:24.625665 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:24.625763 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:24.625765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:24.625799 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:24.625857 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:24.626249 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.626777 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.627472 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:24.627676 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:24.627738 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:24.629735 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:24.629738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:24.629740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:24.629761 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:24.630934 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.631363 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.631366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.631367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.631442 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:24.631675 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:24.631679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:24.631681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:24.631876 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:24.631992 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:24.632265 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 22:27:24.632359 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:24.633380 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23004
I20260618 22:27:24.638152 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23138
I20260618 22:27:24.642483 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23272
I20260618 22:27:24.646529 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 22910
2026-06-18T22:27:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2051 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[23421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[23421](info): set up 2 sockets
Jun 18 22:27:24 dist-test-slave-jm5g krb5kdc[23421](info): commencing operation
krb5kdc: starting...
W20260618 22:27:25.598717 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:25 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab.
Entry for principal oryx/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab.
I20260618 22:27:25.711650 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43117
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab
--principal=oryx/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:25.819401 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:25.819686 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:25.819728 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:25.823750 23434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:25.823813 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:25.823832 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:25.823848 23434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:25.823861 23434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:25.829253 23434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab
--principal=oryx/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:25.830298 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:25.831279 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:25.839300 23439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:25.839352 23442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:25.839325 23440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:25.839673 23434 server_base.cc:1034] running on GCE node
I20260618 22:27:25.840261 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:25.841408 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:25.842604 23434 hybrid_clock.cc:648] HybridClock initialized: now 1781821645842580 us; error 38 us; skew 500 ppm
Jun 18 22:27:25 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:25.845361 23434 init.cc:377] Logged in from keytab as oryx/127.16.243.190@KRBTEST.COM (short username oryx)
I20260618 22:27:25.846262 23434 webserver.cc:460] Webserver started at http://127.16.243.190:34195/ using document root <none> and password file <none>
I20260618 22:27:25.846879 23434 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:25.846946 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:25.847168 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:25.849273 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "71b1e3ffddce4b4da797bdae9199ca49"
format_stamp: "Formatted at 2026-06-18 22:27:25 on dist-test-slave-jm5g"
server_key: "3af7cc0c1b7a97a79c2602e2bc262cac"
server_key_iv: "9c7ca8ddbffd042db43b36913c045d5b"
server_key_version: "encryptionkey@0"
I20260618 22:27:25.849835 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "71b1e3ffddce4b4da797bdae9199ca49"
format_stamp: "Formatted at 2026-06-18 22:27:25 on dist-test-slave-jm5g"
server_key: "3af7cc0c1b7a97a79c2602e2bc262cac"
server_key_iv: "9c7ca8ddbffd042db43b36913c045d5b"
server_key_version: "encryptionkey@0"
I20260618 22:27:25.853605 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:25.856029 23449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:25.857400 23434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:25.857537 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "71b1e3ffddce4b4da797bdae9199ca49"
format_stamp: "Formatted at 2026-06-18 22:27:25 on dist-test-slave-jm5g"
server_key: "3af7cc0c1b7a97a79c2602e2bc262cac"
server_key_iv: "9c7ca8ddbffd042db43b36913c045d5b"
server_key_version: "encryptionkey@0"
I20260618 22:27:25.857638 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:25.876051 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:25.878974 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:25.879166 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:25.885896 23434 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:43117
I20260618 22:27:25.886135 23501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:43117 every 8 connection(s)
I20260618 22:27:25.886924 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:25.887522 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23434
I20260618 22:27:25.887641 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:25.887884 17358 external_mini_cluster.cc:1428] Setting key 10dde6263150bd8db60c28c8960c0686
I20260618 22:27:25.889850 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:25.895143 23502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Bootstrap starting.
Jun 18 22:27:25 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:25.897289 23502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:25.897987 23502 log.cc:824] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:25.900306 23502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: No bootstrap required, opened a new log
I20260618 22:27:25.901204 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:25.889203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.16.243.190:43117)
0618 22:27:25.889596 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:25.889616 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:25.889830 (+ 214us) server_negotiation.cc:407] Connection header received
0618 22:27:25.890593 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:25.890611 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:25.890956 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:25.891257 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:25.892034 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:25.893026 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:25.893833 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:25.894285 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:25.894432 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:25.896785 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:25.896790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:25.896802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:25.896829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:25.898468 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:25.898951 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:25.898956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:25.898960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:25.899071 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:25.899333 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:25.899335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:25.899336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:25.899728 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:25.899864 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:25.900152 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 22:27:25.900450 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260618 22:27:25.903888 23502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } }
I20260618 22:27:25.904106 23502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:25.904163 23502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b1e3ffddce4b4da797bdae9199ca49, State: Initialized, Role: FOLLOWER
I20260618 22:27:25.904560 23502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } }
I20260618 22:27:25.904691 23502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:25.904752 23502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:25.904842 23502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:25.905748 23502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } }
I20260618 22:27:25.906065 23502 leader_election.cc:302] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b1e3ffddce4b4da797bdae9199ca49; no voters:
I20260618 22:27:25.906369 23502 leader_election.cc:288] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:25.906474 23507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:25.906704 23507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: Becoming Leader. State: Replica: 71b1e3ffddce4b4da797bdae9199ca49, State: Running, Role: LEADER
I20260618 22:27:25.906985 23507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } }
I20260618 22:27:25.907509 23502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:25.908589 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } }
I20260618 22:27:25.908715 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:25.909057 23516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:25.909278 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b1e3ffddce4b4da797bdae9199ca49. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } }
I20260618 22:27:25.909374 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:25.912271 23516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:25.917551 23516 catalog_manager.cc:1347] Generated new cluster ID: 7915d150cb624984aa5be62bb4021d82
I20260618 22:27:25.917618 23516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:25.934459 23516 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:25.935590 23516 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:25.942410 23516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Generated new TSK 0
I20260618 22:27:25.943120 23516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
I20260618 22:27:25.977443 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:26.086999 23528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:26.087266 23528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:26.087307 23528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:26.091289 23528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:26.091349 23528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:26.091406 23528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:26.096637 23528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:26.097643 23528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:26.098897 23528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:26.109846 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.109843 23533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.109959 23528 server_base.cc:1034] running on GCE node
W20260618 22:27:26.110093 23534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.110518 23528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:26.111173 23528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:26.112332 23528 hybrid_clock.cc:648] HybridClock initialized: now 1781821646112314 us; error 35 us; skew 500 ppm
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:26.114948 23528 init.cc:377] Logged in from keytab as oryx/127.16.243.129@KRBTEST.COM (short username oryx)
I20260618 22:27:26.115788 23528 webserver.cc:460] Webserver started at http://127.16.243.129:35139/ using document root <none> and password file <none>
I20260618 22:27:26.116320 23528 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.116385 23528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.116569 23528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:26.118657 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "c852ec510b6d419ab1506b753219c924"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "cfc9e65c1648604ed3077c558dfd2a2c"
server_key_iv: "5fbf58e0b514000aa45d73646d54ad12"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.119231 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "c852ec510b6d419ab1506b753219c924"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "cfc9e65c1648604ed3077c558dfd2a2c"
server_key_iv: "5fbf58e0b514000aa45d73646d54ad12"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.123070 23528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:27:26.125449 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.126799 23528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:26.126916 23528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "c852ec510b6d419ab1506b753219c924"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "cfc9e65c1648604ed3077c558dfd2a2c"
server_key_iv: "5fbf58e0b514000aa45d73646d54ad12"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.127027 23528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.141867 23528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:26.145258 23528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:26.145440 23528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:26.146018 23528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:26.146878 23528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:26.146925 23528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.146976 23528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:26.147002 23528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.155056 23528 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:34405
I20260618 22:27:26.155115 23656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:34405 every 8 connection(s)
I20260618 22:27:26.155944 23528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:26.163451 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23528
I20260618 22:27:26.163571 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:26.163885 17358 external_mini_cluster.cc:1428] Setting key e5e3cc763c624a64f92d567fa7d70006
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:26.170892 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.157683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:55663 (local address 127.16.243.190:43117)
0618 22:27:26.157804 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.157807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.158633 (+ 826us) server_negotiation.cc:407] Connection header received
0618 22:27:26.160008 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.160011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.160068 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.160138 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.161405 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.161984 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.163166 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.163498 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.163604 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.166826 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.166831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.166835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.166864 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.168724 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.169218 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.169222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.169223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.169309 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.169610 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.169613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.169615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.169866 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.169968 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.170544 (+ 576us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.170718 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:27:26.171705 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.158032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:55663)
0618 22:27:26.158475 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.158517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.159793 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.160321 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.160328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.160745 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.161258 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.161273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.162100 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.162102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.163045 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.163054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.163694 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.163697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.163868 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.164736 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.164756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.166604 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.168840 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.168845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.168859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.169119 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.169384 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.169387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.169388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.169519 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.169992 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.170000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.170273 (+ 273us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.170457 (+ 184us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.170804 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260618 22:27:26.172979 23657 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:26.173251 23657 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:26.173853 23657 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:26.175459 23466 ts_manager.cc:194] Registered new tserver with Master: c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:26.176877 23466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:55663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
I20260618 22:27:26.191692 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:26.299849 23662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:26.300130 23662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:26.300184 23662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:26.304342 23662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:26.304412 23662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:26.304473 23662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:26.310103 23662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:26.311205 23662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:26.312197 23662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:26.321442 23667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.321512 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.321481 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.321696 23662 server_base.cc:1034] running on GCE node
I20260618 22:27:26.322096 23662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:26.322759 23662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:26.323937 23662 hybrid_clock.cc:648] HybridClock initialized: now 1781821646323920 us; error 36 us; skew 500 ppm
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:26.327083 23662 init.cc:377] Logged in from keytab as oryx/127.16.243.130@KRBTEST.COM (short username oryx)
I20260618 22:27:26.327981 23662 webserver.cc:460] Webserver started at http://127.16.243.130:36559/ using document root <none> and password file <none>
I20260618 22:27:26.328518 23662 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.328583 23662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.328752 23662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:26.330933 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "c9930d5de0b0402f9f1860797eb640e3"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "f15f65eb12e041ee02362072549053c8"
server_key_iv: "2a6e6d3d202f0772357ca8e23c875eed"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.331504 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "c9930d5de0b0402f9f1860797eb640e3"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "f15f65eb12e041ee02362072549053c8"
server_key_iv: "2a6e6d3d202f0772357ca8e23c875eed"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.335315 23662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:27:26.337702 23677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.338970 23662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:26.339082 23662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "c9930d5de0b0402f9f1860797eb640e3"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "f15f65eb12e041ee02362072549053c8"
server_key_iv: "2a6e6d3d202f0772357ca8e23c875eed"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.339179 23662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.360603 23662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:26.363952 23662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:26.364135 23662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:26.364717 23662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:26.365535 23662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:26.365597 23662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.365651 23662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:26.365674 23662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.373977 23662 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:42019
I20260618 22:27:26.373999 23790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:42019 every 8 connection(s)
I20260618 22:27:26.374912 23662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:26.377630 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23662
I20260618 22:27:26.377740 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:26.378010 17358 external_mini_cluster.cc:1428] Setting key db754fc138ca6bc4281c0a587eba79e2
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:26.389156 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.376677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:51867 (local address 127.16.243.190:43117)
0618 22:27:26.376833 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.376837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.377693 (+ 856us) server_negotiation.cc:407] Connection header received
0618 22:27:26.379328 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.379332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.379381 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.379445 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.380619 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.381264 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.382868 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.383082 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.383179 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.385947 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.385951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.385953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.385978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.387329 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.387769 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.387771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.387772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.387840 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.388138 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.388140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.388142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.388332 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.388405 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.388902 (+ 497us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.389035 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:26.389984 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.377029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:51867)
0618 22:27:26.377505 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.377566 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.379084 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.379635 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.379643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.380001 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.380472 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.380485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.381392 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.381395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.382754 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.382765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.383229 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.383233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.383435 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.384032 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.384053 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.385766 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.387429 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.387434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.387447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.387692 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.387935 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.387937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.387938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.388067 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.388409 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.388415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.388652 (+ 237us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.388818 (+ 166us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.389078 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260618 22:27:26.391114 23791 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:26.391361 23791 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:26.391839 23791 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:26.392849 23466 ts_manager.cc:194] Registered new tserver with Master: c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019)
I20260618 22:27:26.393400 23466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
I20260618 22:27:26.403824 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:26.512672 23796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:26.512946 23796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:26.512988 23796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:26.517045 23796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:26.517108 23796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:26.517181 23796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:26.522485 23796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:26.523562 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:26.524600 23796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:26.534884 23801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.534852 23804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.534889 23802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.535511 23796 server_base.cc:1034] running on GCE node
I20260618 22:27:26.535879 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:26.536475 23796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:26.537632 23796 hybrid_clock.cc:648] HybridClock initialized: now 1781821646537602 us; error 40 us; skew 500 ppm
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:26.540277 23796 init.cc:377] Logged in from keytab as oryx/127.16.243.131@KRBTEST.COM (short username oryx)
I20260618 22:27:26.541124 23796 webserver.cc:460] Webserver started at http://127.16.243.131:34221/ using document root <none> and password file <none>
I20260618 22:27:26.541644 23796 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.541704 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.541908 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:26.544046 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "9349c24220adf7c612cea05fed4f436b"
server_key_iv: "a84187f522af673426e58b3a8bc15906"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.544561 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "9349c24220adf7c612cea05fed4f436b"
server_key_iv: "a84187f522af673426e58b3a8bc15906"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.548076 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260618 22:27:26.550370 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.551524 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:26.551640 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "9349c24220adf7c612cea05fed4f436b"
server_key_iv: "a84187f522af673426e58b3a8bc15906"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.551728 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.569098 23796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:26.572067 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:26.572242 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:26.572795 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:26.573581 23796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:26.573627 23796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.573679 23796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:26.573705 23796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.582093 23796 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:38965
I20260618 22:27:26.582114 23924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:38965 every 8 connection(s)
I20260618 22:27:26.583007 23796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:26.590091 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23796
I20260618 22:27:26.590206 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:26.590469 17358 external_mini_cluster.cc:1428] Setting key b963e8680a87ddec38e48a75c7656941
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:26.595533 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.584775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:36455 (local address 127.16.243.190:43117)
0618 22:27:26.584908 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.584911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.585710 (+ 799us) server_negotiation.cc:407] Connection header received
0618 22:27:26.587008 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.587011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.587058 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.587129 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.588236 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.588742 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.589783 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.590058 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.590156 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.592703 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.592706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.592708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.592731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.593867 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.594299 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.594301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.594303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.594371 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.594670 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.594673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.594674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.594838 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.594907 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.595327 (+ 420us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.595416 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:27:26.596321 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.585134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:36455)
0618 22:27:26.585563 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.585605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.586827 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.587296 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.587303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.587654 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.588115 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.588128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.588828 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.588830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.589696 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.589704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.590224 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.590229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.590388 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.590951 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.590967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.592556 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.593969 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.593973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.593986 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.594227 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.594432 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.594435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.594436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.594601 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.594909 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.594914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.595139 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.595287 (+ 148us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.595533 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260618 22:27:26.597302 23925 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:26.597510 23925 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:26.597956 23925 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:26.598871 23466 ts_manager.cc:194] Registered new tserver with Master: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:26.599417 23466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:36455
I20260618 22:27:26.604111 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.189.keytab.
Entry for principal oryx/127.16.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.189.keytab.
I20260618 22:27:26.628916 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:46553
--webserver_interface=127.16.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.189.keytab
--principal=oryx/127.16.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43117,127.16.243.189:46553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:26.738709 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:26.738987 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:26.739032 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:26.743156 23930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:26.743225 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:26.743244 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:26.743260 23930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:26.743275 23930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:26.748772 23930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43117,127.16.243.189:46553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.189.keytab
--principal=oryx/127.16.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:46553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
W20260618 22:27:26.748993 23930 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.243.190:43117,127.16.243.189:46553'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260618 22:27:26.749912 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:26.750989 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:26.759284 23936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.759287 23938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.759317 23935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.759502 23930 server_base.cc:1034] running on GCE node
I20260618 22:27:26.760210 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:26.761404 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:26.762612 23930 hybrid_clock.cc:648] HybridClock initialized: now 1781821646762588 us; error 39 us; skew 500 ppm
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:26.765415 23930 init.cc:377] Logged in from keytab as oryx/127.16.243.189@KRBTEST.COM (short username oryx)
I20260618 22:27:26.766336 23930 webserver.cc:460] Webserver started at http://127.16.243.189:39479/ using document root <none> and password file <none>
I20260618 22:27:26.766907 23930 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.766973 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.767184 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:26.769230 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/instance:
uuid: "13287b61dedb4f3fbe97ed2159a93302"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "09010b905234d2ea4f869651fb08eb1b"
server_key_iv: "eb4394b26e8d90b24f14b399eff68916"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.769754 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal/instance:
uuid: "13287b61dedb4f3fbe97ed2159a93302"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "09010b905234d2ea4f869651fb08eb1b"
server_key_iv: "eb4394b26e8d90b24f14b399eff68916"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.773432 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:27:26.775856 23945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.777042 23930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:26.777165 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "13287b61dedb4f3fbe97ed2159a93302"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "09010b905234d2ea4f869651fb08eb1b"
server_key_iv: "eb4394b26e8d90b24f14b399eff68916"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.777261 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.787650 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:26.790618 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:26.790796 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:26.797204 23930 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.189:46553
I20260618 22:27:26.797425 23997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.189:46553 every 8 connection(s)
I20260618 22:27:26.798144 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
I20260618 22:27:26.800885 23998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:26.804817 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 23930
I20260618 22:27:26.804654 23998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.189@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
I20260618 22:27:26.818415 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.806481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.16.243.190:43117)
0618 22:27:26.806631 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.806636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.807040 (+ 404us) server_negotiation.cc:407] Connection header received
0618 22:27:26.807831 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.807834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.807879 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.807967 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.808980 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.809834 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.811246 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.811633 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.811770 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.814383 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.814386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.814388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.814415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.816005 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.816725 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.816728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.816729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.816839 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.817226 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.817229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.817231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.817530 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.817623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.818122 (+ 499us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.818252 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:27:26.819300 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.806398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47390)
0618 22:27:26.806900 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.806946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.807642 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.808020 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.808025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.808470 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.808847 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.808867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.809947 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.809950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.811108 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.811119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.811799 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.811803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.811950 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.812646 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.812666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.814229 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.816124 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.816133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.816142 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.816577 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.816940 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.816943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.816945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.817119 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.817637 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.817643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.817895 (+ 252us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.818062 (+ 167us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.818400 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"spinlock_wait_cycles":3712,"thread_start_us":157,"threads_started":1}
I20260618 22:27:26.819307 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.806398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.16.243.189:46553)
0618 22:27:26.806863 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.806876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.806908 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:27:26.807633 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.807651 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.807862 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.808024 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.808880 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.810141 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.811288 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.811853 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.811955 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.814872 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.814881 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.814894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.814918 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.816765 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.817272 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.817278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.817279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.817353 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.817580 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.817583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.817585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.817882 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.818031 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.818274 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.818461 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":350,"spinlock_wait_cycles":76160,"thread_start_us":201,"threads_started":1}
I20260618 22:27:26.820634 23998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.189@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
I20260618 22:27:26.828446 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.821098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38178)
0618 22:27:26.821257 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.821271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.821339 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.821662 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.821664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.821793 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.821883 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.821888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.822542 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.822545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.823096 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.823105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.823988 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.823991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.824068 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.824494 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.824504 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.825657 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.827171 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.827174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.827175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.827397 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.827607 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.827609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.827610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.827684 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.827981 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.827983 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.828072 (+ 89us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.828162 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.828276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260618 22:27:26.828474 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.821185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.16.243.189:46553)
0618 22:27:26.821291 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.821294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.821308 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:26.821525 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.821527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.821572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.821632 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.821960 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.822438 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.823201 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.823876 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.823958 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.825792 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.825795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.825797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.825816 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.827061 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.827480 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.827483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.827484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.827545 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.827751 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.827753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.827754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.827904 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.827967 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.828207 (+ 240us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.828327 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":47}
I20260618 22:27:26.831349 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Bootstrap starting.
I20260618 22:27:26.833599 23998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:26.834322 23998 log.cc:824] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:26.836182 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: No bootstrap required, opened a new log
I20260618 22:27:26.839280 23998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } }
I20260618 22:27:26.839486 23998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:26.839529 23998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13287b61dedb4f3fbe97ed2159a93302, State: Initialized, Role: FOLLOWER
I20260618 22:27:26.839951 23998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } }
I20260618 22:27:26.840397 24004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } } }
I20260618 22:27:26.840510 24004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:26.841109 23998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:27:26.845463 24015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:27:26.845543 24015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:27:26.847903 23930 master_runner.cc:428] Detected that this master 13287b61dedb4f3fbe97ed2159a93302 is joining an existing cluster
I20260618 22:27:26.847956 23930 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.243.189:46553
I20260618 22:27:26.857899 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.850221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.16.243.189:46553)
0618 22:27:26.850319 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.850335 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.850765 (+ 430us) server_negotiation.cc:407] Connection header received
0618 22:27:26.850846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.850849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.850891 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.850964 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.851500 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.852227 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.853118 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.853428 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.853524 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.855040 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.855044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.855046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.855067 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.856481 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.856843 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.856846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.856848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.856922 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.857148 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.857151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.857153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.857341 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.857388 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.857615 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.857732 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:27:26.858146 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.849857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.16.243.190:43117)
0618 22:27:26.849973 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.849978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.850144 (+ 166us) server_negotiation.cc:407] Connection header received
0618 22:27:26.850261 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.850265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.850308 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.850372 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.850916 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.851432 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.852242 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.852557 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.852671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.853810 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.853814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.853817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.853846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.855491 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.856917 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.856921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.856922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.857016 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.857279 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.857282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.857284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.857489 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.857580 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.857806 (+ 226us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.857917 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:27:26.858311 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.850394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38190)
0618 22:27:26.850676 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.850688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.850776 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.851053 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.851056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.851265 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.851389 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.851396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.852328 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.852331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.853012 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.853020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.853538 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.853541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.853632 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.854488 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.854499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.854932 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.856598 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.856601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.856603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.856766 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.856993 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.856996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.856998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.857067 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.857427 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.857429 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.857494 (+ 65us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.858052 (+ 558us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.858154 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260618 22:27:26.858448 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.849799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47398)
0618 22:27:26.850068 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.850080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.850183 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.850406 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.850409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.850680 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.850822 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.850829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.851528 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.851532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.852129 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.852136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.852686 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.852690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.852787 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.853301 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.853312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.853676 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.855595 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.855599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.855601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.856020 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.857113 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.857116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.857118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.857201 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.857579 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.857581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.857648 (+ 67us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.857740 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.857851 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.190@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
I20260618 22:27:26.871914 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.862737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.16.243.189:46553)
0618 22:27:26.862864 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.862869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.863245 (+ 376us) server_negotiation.cc:407] Connection header received
0618 22:27:26.863411 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.863413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.863455 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.863517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.864278 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.864773 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.865990 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.866212 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.866290 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.868634 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.868638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.868641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.868673 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.870235 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.870741 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.870744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.870746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.870827 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.871048 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.871051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.871052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.871251 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.871340 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.871631 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.871747 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:27:26.871919 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.862736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38198)
0618 22:27:26.863109 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.863164 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.863297 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.863561 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.863568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.864021 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.864148 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.864156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.864867 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.864870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.865864 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.865883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.866313 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.866317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.866429 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.866922 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.866936 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.868495 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.870344 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.870352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.870357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.870657 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.870905 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.870907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.870909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.870983 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.871333 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.871338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.871441 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.871553 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.871754 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260618 22:27:26.873612 23466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.243.189:46553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302"
member_type: NON_VOTER
last_known_addr {
host: "127.16.243.189"
port: 46553
}
attrs {
promote: true
}
}
dest_uuid: "71b1e3ffddce4b4da797bdae9199ca49"
cas_config_opid_index: -1
I20260618 22:27:26.874295 23466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } }
I20260618 22:27:26.877106 23972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:26.878579 23972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260618 22:27:26.876407 24026 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.16.243.188@KRBTEST.COM; defaulting to no policy
I20260618 22:27:26.880621 24026 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.243.189:46553
I20260618 22:27:26.880949 24026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:26.881040 23930 master.cc:537] Master@127.16.243.189:46553 shutting down...
I20260618 22:27:26.881170 24027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } }
I20260618 22:27:26.881560 24027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.188@KRBTEST.COM" created.
W20260618 22:27:26.885119 23453 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260618 22:27:26.886081 23453 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 -> Peer 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189:46553): Couldn't send request to peer 13287b61dedb4f3fbe97ed2159a93302. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:27:26.886536 23930 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:26.886847 23930 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:26.886948 23930 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: stopping tablet replica
I20260618 22:27:26.891414 23930 master.cc:552] Master@127.16.243.189:46553 shutdown complete.
I20260618 22:27:26.891469 23930 master_runner.cc:305] Clearing existing system tablet
I20260618 22:27:26.891701 23930 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.891752 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.892956 23930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 22:27:26.893782 24032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.894029 23930 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.894146 23930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:26.894177 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "13287b61dedb4f3fbe97ed2159a93302"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "09010b905234d2ea4f869651fb08eb1b"
server_key_iv: "eb4394b26e8d90b24f14b399eff68916"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.896224 23930 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:26.898610 23930 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:27:26.898716 23930 log.cc:1196] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260618 22:27:26.898957 23930 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Deleting consensus metadata
I20260618 22:27:26.899111 23930 master_runner.cc:315] Copying system tablet from 127.16.243.190:43117
I20260618 22:27:26.900298 23930 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: tablet copy: Beginning tablet copy session from remote peer at address 127.16.243.190:43117
I20260618 22:27:26.909282 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.900906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47406)
0618 22:27:26.901219 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:26.901235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:26.901362 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:26.901636 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:26.901638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:26.901855 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:26.901978 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.901987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.903141 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.903145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.904095 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:26.904107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.904695 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.904699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:26.904790 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.905343 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:26.905359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:26.905793 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:26.907700 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.907704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.907706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.907931 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.908222 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:26.908225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:26.908228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:26.908306 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:26.908728 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:26.908731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:26.908831 (+ 100us) client_negotiation.cc:769] Sending connection context
0618 22:27:26.908927 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 22:27:26.909074 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
I20260618 22:27:26.909272 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:26.900992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.16.243.190:43117)
0618 22:27:26.901114 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:26.901119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:26.901321 (+ 202us) server_negotiation.cc:407] Connection header received
0618 22:27:26.901455 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:26.901458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:26.901511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:26.901606 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:26.902122 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.903018 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.904221 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:26.904563 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:26.904674 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:26.905955 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:26.905959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:26.905962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:26.905990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:26.907602 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.908029 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.908046 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.908048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.908127 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:26.908387 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:26.908390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:26.908392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:26.908633 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:26.908726 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:26.908990 (+ 264us) server_negotiation.cc:299] Negotiation successful
0618 22:27:26.909098 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:27:26.909915 23486 tablet_copy_service.cc:140] P 71b1e3ffddce4b4da797bdae9199ca49: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13287b61dedb4f3fbe97ed2159a93302 ({username='oryx', principal='oryx/127.16.243.189@KRBTEST.COM'} at 127.0.0.1:47406)
I20260618 22:27:26.910070 23486 tablet_copy_service.cc:161] P 71b1e3ffddce4b4da797bdae9199ca49: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13287b61dedb4f3fbe97ed2159a93302 at {username='oryx', principal='oryx/127.16.243.189@KRBTEST.COM'} at 127.0.0.1:47406: session id = 13287b61dedb4f3fbe97ed2159a93302-00000000000000000000000000000000
I20260618 22:27:26.911639 23486 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.188.keytab.
Entry for principal oryx/127.16.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.188.keytab.
I20260618 22:27:26.913204 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:39941
--webserver_interface=127.16.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.188.keytab
--principal=oryx/127.16.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:27:26.913300 23930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:27:26.915654 23930 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: tablet copy: Starting download of 0 data blocks...
I20260618 22:27:26.915863 23930 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: tablet copy: Starting download of 1 WAL segments...
I20260618 22:27:26.917754 23930 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:27:26.919797 23486 tablet_copy_service.cc:342] P 71b1e3ffddce4b4da797bdae9199ca49: Request end of tablet copy session 13287b61dedb4f3fbe97ed2159a93302-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.243.189@KRBTEST.COM'} at 127.0.0.1:47406
I20260618 22:27:26.919941 23486 tablet_copy_service.cc:433] P 71b1e3ffddce4b4da797bdae9199ca49: ending tablet copy session 13287b61dedb4f3fbe97ed2159a93302-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13287b61dedb4f3fbe97ed2159a93302
I20260618 22:27:26.927415 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:26.929075 24040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.931515 24041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:26.931620 24043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:26.934718 23930 server_base.cc:1034] running on GCE node
I20260618 22:27:26.934897 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:26.935200 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:26.936342 23930 hybrid_clock.cc:648] HybridClock initialized: now 1781821646936322 us; error 38 us; skew 500 ppm
Jun 18 22:27:26 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821646, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:26.937901 23930 init.cc:377] Logged in from keytab as oryx/127.16.243.189@KRBTEST.COM (short username oryx)
I20260618 22:27:26.938354 23930 webserver.cc:460] Webserver started at http://127.16.243.189:33927/ using document root <none> and password file <none>
I20260618 22:27:26.938532 23930 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:26.938608 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:26.939877 23930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:26.940806 24050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:26.941028 23930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:26.941103 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "13287b61dedb4f3fbe97ed2159a93302"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "09010b905234d2ea4f869651fb08eb1b"
server_key_iv: "eb4394b26e8d90b24f14b399eff68916"
server_key_version: "encryptionkey@0"
I20260618 22:27:26.941192 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:26.948941 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:26.951004 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:26.955565 23930 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.189:46553
I20260618 22:27:26.956204 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
I20260618 22:27:26.956566 24102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.189:46553 every 8 connection(s)
I20260618 22:27:26.958046 24103 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:27:26.959124 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Bootstrap starting.
I20260618 22:27:26.969774 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:27:26.970276 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: Bootstrap complete.
I20260618 22:27:26.970985 24103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } }
I20260618 22:27:26.971313 24103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13287b61dedb4f3fbe97ed2159a93302, State: Initialized, Role: LEARNER
I20260618 22:27:26.971489 24103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } }
I20260618 22:27:26.971963 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } }
I20260618 22:27:26.972075 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:27:26.972177 24103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:26.975116 24117 catalog_manager.cc:1259] Loaded cluster ID: 7915d150cb624984aa5be62bb4021d82
I20260618 22:27:26.975188 24117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: loading cluster ID for follower catalog manager: success
I20260618 22:27:26.977209 24117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: acquiring CA information for follower catalog manager: success
I20260618 22:27:26.978603 24117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:27:27.035670 24039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:27.035955 24039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:27.035996 24039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:27.039994 24039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:27.040055 24039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:27.040072 24039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:27.040087 24039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:27.040108 24039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:27.045562 24039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.188.keytab
--principal=oryx/127.16.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:27.046638 24039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:27.047631 24039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:27.055624 24122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.055791 24039 server_base.cc:1034] running on GCE node
W20260618 22:27:27.055644 24123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.055635 24125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.056489 24039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:27.057548 24039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:27.058717 24039 hybrid_clock.cc:648] HybridClock initialized: now 1781821647058691 us; error 36 us; skew 500 ppm
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:27.061406 24039 init.cc:377] Logged in from keytab as oryx/127.16.243.188@KRBTEST.COM (short username oryx)
I20260618 22:27:27.062341 24039 webserver.cc:460] Webserver started at http://127.16.243.188:37641/ using document root <none> and password file <none>
I20260618 22:27:27.062911 24039 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.062978 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.063185 24039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:27.065186 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/instance:
uuid: "3abdcce626814229bf33fd386366766c"
format_stamp: "Formatted at 2026-06-18 22:27:27 on dist-test-slave-jm5g"
server_key: "4d771c06747cccd0c6ffbad614afc84b"
server_key_iv: "0618b0ac9c49980e1fc9e4aa95bd4539"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.065706 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal/instance:
uuid: "3abdcce626814229bf33fd386366766c"
format_stamp: "Formatted at 2026-06-18 22:27:27 on dist-test-slave-jm5g"
server_key: "4d771c06747cccd0c6ffbad614afc84b"
server_key_iv: "0618b0ac9c49980e1fc9e4aa95bd4539"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.069397 24039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:27.071789 24132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.073189 24039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:27.073313 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "3abdcce626814229bf33fd386366766c"
format_stamp: "Formatted at 2026-06-18 22:27:27 on dist-test-slave-jm5g"
server_key: "4d771c06747cccd0c6ffbad614afc84b"
server_key_iv: "0618b0ac9c49980e1fc9e4aa95bd4539"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.073413 24039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.090124 24039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:27.092936 24039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:27.093099 24039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:27.099648 24039 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.188:39941
I20260618 22:27:27.099886 24184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.188:39941 every 8 connection(s)
I20260618 22:27:27.100603 24039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
I20260618 22:27:27.103319 24185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:27.106925 24185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:27:27.109505 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24039
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.188@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
I20260618 22:27:27.119905 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.108895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.16.243.190:43117)
0618 22:27:27.109062 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.109067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.109931 (+ 864us) server_negotiation.cc:407] Connection header received
0618 22:27:27.111051 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.111054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.111099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.111162 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.112034 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.112539 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.113604 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.113840 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.113913 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.116484 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.116487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.116489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.116513 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.117958 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.118438 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.118441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.118443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.118537 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.118870 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.118874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.118876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.119054 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.119141 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.119623 (+ 482us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.119742 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:27:27.120782 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.110216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.16.243.188:39941)
0618 22:27:27.110478 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.110489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.110577 (+ 88us) server_negotiation.cc:407] Connection header received
0618 22:27:27.110857 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.110877 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.111102 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.111268 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.111889 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.112840 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.113628 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.114039 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.114130 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.116872 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.116879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.116892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.116916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.118458 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.118923 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.118928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.118930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.119006 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.119253 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.119257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.119258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.119479 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.119661 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.119865 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.119995 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260618 22:27:27.120780 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.109259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47418)
0618 22:27:27.109763 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.109819 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.110849 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.111228 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.111233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.111630 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.111897 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.111910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.112655 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.112658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.113484 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.113493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.113967 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.113971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.114114 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.114882 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.114897 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.116337 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.118056 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.118063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.118072 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.118346 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.118644 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.118647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.118650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.118784 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.119155 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.119162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.119399 (+ 237us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.119562 (+ 163us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.119877 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":40,"thread_start_us":116,"threads_started":1}
I20260618 22:27:27.122243 24185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.188@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
I20260618 22:27:27.129562 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.122717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38210)
0618 22:27:27.122996 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.123010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.123092 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.123329 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.123331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.123478 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.123569 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.123574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.124449 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.124453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.124983 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.124990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.125409 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.125412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.125503 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.125877 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.125886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.127025 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.128438 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.128440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.128441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.128614 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.128811 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.128813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.128814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.128896 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.129199 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.129201 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.129278 (+ 77us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.129350 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.129446 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260618 22:27:27.129621 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.122799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.16.243.189:46553)
0618 22:27:27.123013 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.123018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.123053 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:27:27.123174 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.123191 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.123237 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.123307 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.123670 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.124354 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.125083 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.125319 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.125402 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.127140 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.127143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.127145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.127168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.128342 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.128681 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.128684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.128686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.128753 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.128961 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.128963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.128964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.129128 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.129208 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.129390 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.129488 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260618 22:27:27.130282 24185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.188@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
I20260618 22:27:27.137122 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.130731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:57850)
0618 22:27:27.130822 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.130836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.130907 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.131172 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.131189 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.131311 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.131374 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.131379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.132087 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.132089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.132586 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.132594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.132990 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.132992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.133044 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.133461 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.133472 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.134528 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.136027 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.136029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.136031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.136211 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.136398 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.136400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.136401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.136462 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.136742 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.136744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.136813 (+ 69us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.136883 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.136983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260618 22:27:27.137151 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.130776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.16.243.188:39941)
0618 22:27:27.130871 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.130874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.130887 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:27.130972 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.130975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.131007 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.131071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.131472 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.131984 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.132684 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.132910 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.132985 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.134712 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.134715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.134718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.134740 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.135934 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.136279 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.136281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.136282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.136342 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.136517 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.136519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.136520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.136666 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.136746 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.136923 (+ 177us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.137034 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":40}
I20260618 22:27:27.140336 24185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Bootstrap starting.
I20260618 22:27:27.142719 24185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:27.143419 24185 log.cc:824] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:27.145268 24185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: No bootstrap required, opened a new log
I20260618 22:27:27.148214 24185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } }
I20260618 22:27:27.148418 24185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:27.148463 24185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abdcce626814229bf33fd386366766c, State: Initialized, Role: FOLLOWER
I20260618 22:27:27.148833 24185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } }
I20260618 22:27:27.149236 24191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } } }
I20260618 22:27:27.149356 24191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:27.149946 24185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:27:27.154337 24202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:27:27.154423 24202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:27:27.157872 24039 master_runner.cc:428] Detected that this master 3abdcce626814229bf33fd386366766c is joining an existing cluster
I20260618 22:27:27.157935 24039 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.243.188:39941
I20260618 22:27:27.174387 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23528
I20260618 22:27:27.180109 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:34405
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:43117
--webserver_port=35139
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:27:27.190774 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.16.243.188:39941)
0618 22:27:27.160822 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.160827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.160840 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:27.160875 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.160877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.160920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.160994 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.161635 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.162414 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.163566 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.163881 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.163942 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.165206 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.165210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.165213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.165237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.188418 (+ 23181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.189231 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.189235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.189237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.189566 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.189854 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.189857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.189859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.190074 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.190140 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.190454 (+ 314us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.190604 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":57}
I20260618 22:27:27.191283 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:57862)
0618 22:27:27.160571 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.160585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.160673 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.161135 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.161138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.161409 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.161540 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.161548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.162597 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.162600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.163465 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.163473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.164078 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.164157 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.164702 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.164714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.165084 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.188560 (+ 23476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.188563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.188565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.189107 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.189674 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.189676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.189678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.189766 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.190206 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.190210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.190305 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.190986 (+ 681us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.191121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260618 22:27:27.192332 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38222)
0618 22:27:27.161121 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.161133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.161273 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.161584 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.161587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.161778 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.161887 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.161894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.162866 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.162869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.163689 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.163696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.164341 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.164420 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.187371 (+ 22951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.187620 (+ 249us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.188194 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.190241 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.190244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.190247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.190479 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.190843 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.190846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.190848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.190931 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.191836 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.191839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.191932 (+ 93us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.192027 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.192156 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":3072,"thread_start_us":77,"threads_started":1}
I20260618 22:27:27.193082 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.16.243.189:46553)
0618 22:27:27.160875 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.160879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.161205 (+ 326us) server_negotiation.cc:407] Connection header received
0618 22:27:27.161372 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.161374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.161419 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.161466 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.162007 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.162748 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.163809 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.164123 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.188439 (+ 24239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.188444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.188447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.188478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.190110 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.190615 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.190619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.190621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.190729 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.191268 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.191271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.191273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.191485 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.191571 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.192796 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.192928 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 22:27:27.195689 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47420)
0618 22:27:27.161115 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.161128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.161271 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.161657 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.161660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.161904 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.162018 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.162026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.163110 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.163113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.164061 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.164071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.164754 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.164845 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.165331 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.165342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.191854 (+ 26512us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x2b1786b8. stack: 00007fdc0ce39c78 00007fdc0ce3a1a9 00007fdc0c9fc1d7 00000000005984a6 00000000005984f9 000000000059c488 00007fdc0ce362ab 00007fdc0ce35ad0 00007fdc1150e0ee 00007fdc11490cae 00007fdc1149007b 00007fdc1148ce17 00007fdc114cd832 00007fdc114ce68c 00007fdc114e40d3 00007fdc114e754b
0618 22:27:27.191885 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.193921 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.193924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.193926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.194143 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.194464 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.194466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.194468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.194745 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.195165 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.195174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.195270 (+ 96us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.195367 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.195503 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":25,"spinlock_wait_cycles":4763648,"thread_start_us":80,"threads_started":1}
I20260618 22:27:27.195703 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.160340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.16.243.190:43117)
0618 22:27:27.160460 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.160465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.161194 (+ 729us) server_negotiation.cc:407] Connection header received
0618 22:27:27.161373 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.161377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.161424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.161490 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.162152 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.162974 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164193 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.164521 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.164597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.192064 (+ 27467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.192068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.192071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.192103 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.193726 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.194264 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.194268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.194270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.194366 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.194839 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.194842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.194844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.195071 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.195170 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.195427 (+ 257us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.195543 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821645, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.190@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
I20260618 22:27:27.208173 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.199420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:57870)
0618 22:27:27.199536 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.199551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.199630 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.199899 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.199902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.200089 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.200212 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.200219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.201159 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.201162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.202022 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.202033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.202673 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.202685 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.202782 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.203252 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.203266 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.204818 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.206592 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.206595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.206597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.206838 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.207101 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.207104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.207106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.207222 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.207633 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.207636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.207746 (+ 110us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.207852 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.208000 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 22:27:27.208180 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.199494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.16.243.188:39941)
0618 22:27:27.199592 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.199596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.199607 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:27:27.199748 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.199751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.199800 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.199868 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.200318 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.201055 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.202155 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.202488 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.202621 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.204955 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.204959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.204962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.204988 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.206459 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.206931 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.206934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.206936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.207017 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.207325 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.207329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.207331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.207537 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.207628 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.207908 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.208017 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:27.209873 23466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.243.188:39941: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3abdcce626814229bf33fd386366766c"
member_type: NON_VOTER
last_known_addr {
host: "127.16.243.188"
port: 39941
}
attrs {
promote: true
}
}
dest_uuid: "71b1e3ffddce4b4da797bdae9199ca49"
cas_config_opid_index: 5
I20260618 22:27:27.210469 23466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } }
I20260618 22:27:27.211561 24025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3abdcce626814229bf33fd386366766c (127.16.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.212301 24159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:27.213076 24025 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.243.188:39941
I20260618 22:27:27.213263 24025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.213366 24025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:27.213649 24159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 22:27:27.214296 24039 master.cc:537] Master@127.16.243.188:39941 shutting down...
I20260618 22:27:27.214315 24027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:27:27.215977 23450 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 -> Peer 3abdcce626814229bf33fd386366766c (127.16.243.188:39941): Couldn't send request to peer 3abdcce626814229bf33fd386366766c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:27:27.218999 24039 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:27.219270 24039 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:27.219370 24039 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: stopping tablet replica
I20260618 22:27:27.221278 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.211891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:38236)
0618 22:27:27.212343 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.212357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.212472 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.216673 (+ 4201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.216688 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.216706 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.216799 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.216805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.217831 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.217835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.219566 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.219578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.220842 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.220846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.220936 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.220985 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.221055 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.221137 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260618 22:27:27.221632 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.212119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.16.243.189:46553)
0618 22:27:27.216151 (+ 4032us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.216156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.216170 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:27.216219 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.216222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.216280 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.216365 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.216926 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.217713 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.219718 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.220509 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.220637 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.220777 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.221435 (+ 658us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.221516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3939}
I20260618 22:27:27.222343 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 LEARNER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 22:27:27.223093 24027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260618 22:27:27.224900 24039 master.cc:552] Master@127.16.243.188:39941 shutdown complete.
I20260618 22:27:27.224946 24039 master_runner.cc:305] Clearing existing system tablet
I20260618 22:27:27.225167 24039 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.225211 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.224987 24077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3abdcce626814229bf33fd386366766c (127.16.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.226848 24039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 22:27:27.227244 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b1e3ffddce4b4da797bdae9199ca49. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.227347 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:27:27.227435 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: NON_VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: true } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.227499 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:27:27.227605 24215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.227777 24039 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.227881 24039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:27.227916 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "3abdcce626814229bf33fd386366766c"
format_stamp: "Formatted at 2026-06-18 22:27:27 on dist-test-slave-jm5g"
server_key: "4d771c06747cccd0c6ffbad614afc84b"
server_key_iv: "0618b0ac9c49980e1fc9e4aa95bd4539"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.229862 24039 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:27.232219 24039 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:27:27.232313 24039 log.cc:1196] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260618 22:27:27.232555 24039 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Deleting consensus metadata
I20260618 22:27:27.232719 24039 master_runner.cc:315] Copying system tablet from 127.16.243.190:43117
I20260618 22:27:27.233834 24039 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.243.190:43117
I20260618 22:27:27.233968 24027 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: attempting to promote NON_VOTER 13287b61dedb4f3fbe97ed2159a93302 to VOTER
I20260618 22:27:27.234470 24027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } }
I20260618 22:27:27.236023 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.235380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:57886)
0618 22:27:27.235826 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.235923 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:39941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
I20260618 22:27:27.236686 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 LEARNER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 22:27:27.237038 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.236789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:57892)
0618 22:27:27.236919 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.236970 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:39941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260618 22:27:27.237161 24026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260618 22:27:27.237490 23450 consensus_peers.cc:487] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 -> Peer 3abdcce626814229bf33fd386366766c (127.16.243.188:39941): Couldn't send request to peer 3abdcce626814229bf33fd386366766c. Status: Network error: Client connection negotiation failed: client connection to 127.16.243.188:39941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 22:27:27.238919 24025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.239421 24077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.240387 24025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.240484 24025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:27.240589 24025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.240648 24025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:27.241139 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.241271 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:27.248916 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.234860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:47424)
0618 22:27:27.237846 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.237861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.238002 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.238370 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.238373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.238619 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.238730 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.238737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.242642 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.242646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.243574 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.243587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.244203 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.244207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.244294 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.244950 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.244966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.245398 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.247310 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.247313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.247316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.247565 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.247854 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.247857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.247859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.247954 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.248385 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.248388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.248497 (+ 109us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.248595 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.248740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2900,"thread_start_us":64,"threads_started":1}
I20260618 22:27:27.248952 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.234634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.16.243.190:43117)
0618 22:27:27.234767 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.234771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.237927 (+ 3156us) server_negotiation.cc:407] Connection header received
0618 22:27:27.238101 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.238104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.238148 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.238217 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.241596 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.242450 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.243712 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.244070 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.244174 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.245559 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.245563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.245566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.245594 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.247195 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.247668 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.247671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.247673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.247770 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.248068 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.248071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.248072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.248291 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.248382 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.248670 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.248797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:27:27.249518 23486 tablet_copy_service.cc:140] P 71b1e3ffddce4b4da797bdae9199ca49: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3abdcce626814229bf33fd386366766c ({username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:47424)
I20260618 22:27:27.249625 23486 tablet_copy_service.cc:161] P 71b1e3ffddce4b4da797bdae9199ca49: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3abdcce626814229bf33fd386366766c at {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:47424: session id = 3abdcce626814229bf33fd386366766c-00000000000000000000000000000000
I20260618 22:27:27.250835 23486 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:27:27.252156 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:27:27.254522 24039 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: tablet copy: Starting download of 0 data blocks...
I20260618 22:27:27.254907 24039 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: tablet copy: Starting download of 1 WAL segments...
I20260618 22:27:27.256541 24039 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:27:27.258467 23486 tablet_copy_service.cc:342] P 71b1e3ffddce4b4da797bdae9199ca49: Request end of tablet copy session 3abdcce626814229bf33fd386366766c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:47424
I20260618 22:27:27.258613 23486 tablet_copy_service.cc:433] P 71b1e3ffddce4b4da797bdae9199ca49: ending tablet copy session 3abdcce626814229bf33fd386366766c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3abdcce626814229bf33fd386366766c
I20260618 22:27:27.265538 24039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:27.267204 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.267211 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.267462 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.267925 24039 server_base.cc:1034] running on GCE node
I20260618 22:27:27.268106 24039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:27.268399 24039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:27.269541 24039 hybrid_clock.cc:648] HybridClock initialized: now 1781821647269530 us; error 29 us; skew 500 ppm
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:27.271107 24039 init.cc:377] Logged in from keytab as oryx/127.16.243.188@KRBTEST.COM (short username oryx)
I20260618 22:27:27.271571 24039 webserver.cc:460] Webserver started at http://127.16.243.188:45583/ using document root <none> and password file <none>
I20260618 22:27:27.271763 24039 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.271804 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.273098 24039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:27.274109 24232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.274466 24039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:27.274581 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "3abdcce626814229bf33fd386366766c"
format_stamp: "Formatted at 2026-06-18 22:27:27 on dist-test-slave-jm5g"
server_key: "4d771c06747cccd0c6ffbad614afc84b"
server_key_iv: "0618b0ac9c49980e1fc9e4aa95bd4539"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.274688 24039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.293883 24039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:27.296247 24039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:27.301641 24039 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.188:39941
I20260618 22:27:27.301667 24284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.188:39941 every 8 connection(s)
I20260618 22:27:27.302284 24039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
I20260618 22:27:27.304276 24285 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:27:27.305459 24285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Bootstrap starting.
I20260618 22:27:27.317252 24285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:27:27.317710 24285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Bootstrap complete.
I20260618 22:27:27.318420 24285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } }
I20260618 22:27:27.318707 24285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3abdcce626814229bf33fd386366766c, State: Initialized, Role: LEARNER
I20260618 22:27:27.318874 24285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } }
I20260618 22:27:27.319454 24285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:27.319506 24291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.319633 24291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: LEARNER
I20260618 22:27:27.322748 24302 catalog_manager.cc:1259] Loaded cluster ID: 7915d150cb624984aa5be62bb4021d82
I20260618 22:27:27.322829 24302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: loading cluster ID for follower catalog manager: success
I20260618 22:27:27.324872 24302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: acquiring CA information for follower catalog manager: success
W20260618 22:27:27.325197 24211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:27.325493 24211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:27.325560 24211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260618 22:27:27.328419 24302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:27:27.330794 24211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:27.330849 24211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:27.330929 24211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:27.336375 24211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:34405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=35139
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:27.337493 24211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:27.338640 24211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:27.348373 24211 server_base.cc:1034] running on GCE node
W20260618 22:27:27.348279 24307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.348306 24305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.348312 24304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.348973 24211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:27.349620 24211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:27.350762 24211 hybrid_clock.cc:648] HybridClock initialized: now 1781821647350737 us; error 42 us; skew 500 ppm
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:27.353363 24211 init.cc:377] Logged in from keytab as oryx/127.16.243.129@KRBTEST.COM (short username oryx)
I20260618 22:27:27.354259 24211 webserver.cc:460] Webserver started at http://127.16.243.129:35139/ using document root <none> and password file <none>
I20260618 22:27:27.354828 24211 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.354910 24211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.358819 24211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:27.360885 24314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.362213 24211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:27.362321 24211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "c852ec510b6d419ab1506b753219c924"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "cfc9e65c1648604ed3077c558dfd2a2c"
server_key_iv: "5fbf58e0b514000aa45d73646d54ad12"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.362918 24211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.380817 24211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:27.384075 24211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:27.384254 24211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:27.384845 24211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:27.385687 24211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:27.385733 24211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.385775 24211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:27.385803 24211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.394870 24211 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:34405
I20260618 22:27:27.394915 24427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:34405 every 8 connection(s)
I20260618 22:27:27.395450 23791 heartbeater.cc:503] Master 127.16.243.190:43117 was elected leader, sending a full tablet report...
I20260618 22:27:27.395941 24211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:27.399955 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24211
I20260618 22:27:27.400137 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23662
I20260618 22:27:27.406659 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:42019
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:43117
--webserver_port=36559
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:27.412441 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.397722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:41545 (local address 127.16.243.188:39941)
0618 22:27:27.397963 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.397966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.398973 (+ 1007us) server_negotiation.cc:407] Connection header received
0618 22:27:27.400169 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.400173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.400231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.400298 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.401573 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.402676 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404149 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.404482 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404551 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.408245 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.408249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.408252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.408278 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.410056 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.410647 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.410651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.410653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.410744 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411149 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.411153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.411155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.411377 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.411448 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.412124 (+ 676us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.412259 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260618 22:27:27.413024 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.397720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:45571 (local address 127.16.243.189:46553)
0618 22:27:27.397847 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.397851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.398973 (+ 1122us) server_negotiation.cc:407] Connection header received
0618 22:27:27.400133 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.400137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.400212 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.400264 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.401570 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.402369 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404164 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.404521 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404609 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.409196 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.409200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.409203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.409229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.411316 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411798 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.411801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.411803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.411888 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.412167 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.412170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.412171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.412372 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.412446 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.412765 (+ 319us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.412873 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:27:27.413216 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.399592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:32873 (local address 127.16.243.190:43117)
0618 22:27:27.399716 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.399759 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.399774 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:27.404365 (+ 4591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.404369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.404420 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.404472 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.405069 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.405732 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.406817 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.407117 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.407219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.409545 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.409549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.409551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.409578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.411369 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411895 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.411899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.411901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.411991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.412308 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.412312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.412314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.412526 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.412599 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.412926 (+ 327us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.413048 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:27:27.414077 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.398280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.16.243.129:41545)
0618 22:27:27.398812 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.398872 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.399902 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.400568 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.400578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.400937 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.401425 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.401437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.402786 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.402790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.404012 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.404022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.404759 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.404993 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.405677 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.405711 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.408076 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.410167 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.410173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.410192 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.410516 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.410861 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.410864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.410866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.411036 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.411508 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.411514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.411855 (+ 341us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.412639 (+ 784us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.412990 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":36,"spinlock_wait_cycles":65920,"thread_start_us":163,"threads_started":1}
I20260618 22:27:27.414105 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.399362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:32873)
0618 22:27:27.399595 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.399609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.404247 (+ 4638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.404602 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.404605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.404820 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.404935 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.404941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.405868 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.405871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.406680 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.406691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.407340 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.407342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.407404 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.408027 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.408042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.409405 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.411492 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.411497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.411779 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.412102 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.412105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.412106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.412205 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.412662 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.412665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.412750 (+ 85us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.413392 (+ 642us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.413527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260618 22:27:27.414333 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.398280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.16.243.129:45571)
0618 22:27:27.398812 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.398872 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.399898 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.401002 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.401004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.401207 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.401425 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.401439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.402510 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.402513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.404027 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.404037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.404762 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.404765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.404991 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.405720 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.405735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.408707 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.411424 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.411428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.411693 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.411985 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.411988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.411989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.412079 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.412502 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.412505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.412590 (+ 85us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.413209 (+ 619us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.413333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":183,"spinlock_wait_cycles":7808,"thread_start_us":128,"threads_started":1}
I20260618 22:27:27.415577 24429 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:27.415640 24430 heartbeater.cc:347] Connected to a master server at 127.16.243.189:46553
I20260618 22:27:27.415831 24430 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.415859 24428 heartbeater.cc:347] Connected to a master server at 127.16.243.188:39941
I20260618 22:27:27.415832 24429 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.415922 24428 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.416427 24428 heartbeater.cc:511] Master 127.16.243.188:39941 requested a full tablet report, sending...
I20260618 22:27:27.416421 24429 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:27.416422 24430 heartbeater.cc:511] Master 127.16.243.189:46553 requested a full tablet report, sending...
I20260618 22:27:27.417891 23466 ts_manager.cc:194] Re-registered known tserver with Master: c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
I20260618 22:27:27.418049 24249 ts_manager.cc:194] Registered new tserver with Master: c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
I20260618 22:27:27.418048 24059 ts_manager.cc:194] Registered new tserver with Master: c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
I20260618 22:27:27.418699 23466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:32873
W20260618 22:27:27.521111 24436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:27.521420 24436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:27.521479 24436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:27.525694 24436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:27.525764 24436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:27.525826 24436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:27.531312 24436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:42019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=36559
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:27.532532 24436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:27.533589 24436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:27.543296 24444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.543296 24441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.543323 24442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.543522 24436 server_base.cc:1034] running on GCE node
I20260618 22:27:27.544056 24436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:27.544745 24436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:27.545894 24436 hybrid_clock.cc:648] HybridClock initialized: now 1781821647545872 us; error 34 us; skew 500 ppm
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:27.548568 24436 init.cc:377] Logged in from keytab as oryx/127.16.243.130@KRBTEST.COM (short username oryx)
I20260618 22:27:27.549461 24436 webserver.cc:460] Webserver started at http://127.16.243.130:36559/ using document root <none> and password file <none>
I20260618 22:27:27.550004 24436 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.550087 24436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.554041 24436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:27.556275 24451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.557552 24436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:27.557662 24436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "c9930d5de0b0402f9f1860797eb640e3"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "f15f65eb12e041ee02362072549053c8"
server_key_iv: "2a6e6d3d202f0772357ca8e23c875eed"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.558213 24436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.575757 24436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:27.578691 24436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:27.578874 24436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:27.579490 24436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:27.580355 24436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:27.580403 24436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.580454 24436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:27.580482 24436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.589530 24436 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:42019
I20260618 22:27:27.589557 24564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:42019 every 8 connection(s)
I20260618 22:27:27.590507 24436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:27.593662 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24436
I20260618 22:27:27.593848 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23796
I20260618 22:27:27.600625 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:38965
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:43117
--webserver_port=34221
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:27.608518 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.594264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55175 (local address 127.16.243.189:46553)
0618 22:27:27.594370 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.594374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.595599 (+ 1225us) server_negotiation.cc:407] Connection header received
0618 22:27:27.595747 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.595750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.595798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.595848 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.596618 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597468 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599491 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.599834 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599907 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.604588 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.604592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.604595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.604623 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.606435 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.606918 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.606921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.606923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.606999 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.607282 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.607285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.607287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.607506 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.607578 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.608220 (+ 642us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.608346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:27:27.608532 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.592438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:58631 (local address 127.16.243.188:39941)
0618 22:27:27.592536 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.592539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.593492 (+ 953us) server_negotiation.cc:407] Connection header received
0618 22:27:27.594671 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.594675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.594730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.594779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.596696 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597525 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599502 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.599839 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599913 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.603653 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.603657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.603659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.603688 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.605377 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.606598 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.606602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.606604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.606686 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.607091 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.607095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.607096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.607358 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.607437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.608224 (+ 787us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.608351 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:27:27.609232 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.592283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:34209 (local address 127.16.243.190:43117)
0618 22:27:27.592427 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.592431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.594501 (+ 2070us) server_negotiation.cc:407] Connection header received
0618 22:27:27.595180 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.595183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.595230 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.595299 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.596725 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597480 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599491 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.599835 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.599907 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.604285 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.604290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.604293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.604326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.606611 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.607661 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.607665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.607666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.607763 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.608089 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.608092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.608094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.608326 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.608417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.608938 (+ 521us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.609063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:27.610330 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.593938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.16.243.130:55175)
0618 22:27:27.595517 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.595530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.595654 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.595969 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.595971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.596154 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.596459 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.596477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597647 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.597650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.599352 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.599363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.600105 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.600109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.600396 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.601119 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.601147 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.604464 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.606534 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.606538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.606540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.606814 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.607095 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.607098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.607099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.607173 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.607635 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.607638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.607929 (+ 291us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.609091 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.609216 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":336,"spinlock_wait_cycles":34816,"thread_start_us":75,"threads_started":1}
I20260618 22:27:27.610332 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.592785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:34209)
0618 22:27:27.594392 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.594405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.594502 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.595333 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.595336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.595735 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.596471 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.596477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597647 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.597650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.599363 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.599369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.600105 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.600109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.600399 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.601195 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.601207 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.604129 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.606741 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.606744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.606747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.607533 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.607866 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.607869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.607871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.607986 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.608460 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.608463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.608563 (+ 100us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.608675 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.609550 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"mutex_wait_us":1101,"spinlock_wait_cycles":89600,"thread_start_us":161,"threads_started":1}
I20260618 22:27:27.610330 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.592785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.16.243.130:58631)
0618 22:27:27.593332 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.593382 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.594422 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.595087 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.595095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.596087 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.596594 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.596600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.597679 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.597683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.599364 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.599375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.600105 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.600108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.600368 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.601118 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.601143 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.603339 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.605488 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.605494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.605513 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.605862 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.606794 (+ 932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.606797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.606799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.606986 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.607523 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.607530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.607929 (+ 399us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.608762 (+ 833us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.609105 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":406,"spinlock_wait_cycles":75264,"thread_start_us":655,"threads_started":1}
I20260618 22:27:27.611676 24566 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:27.611749 24567 heartbeater.cc:347] Connected to a master server at 127.16.243.189:46553
I20260618 22:27:27.611953 24567 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.611953 24566 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.612237 24565 heartbeater.cc:347] Connected to a master server at 127.16.243.188:39941
I20260618 22:27:27.612313 24565 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.612573 24565 heartbeater.cc:511] Master 127.16.243.188:39941 requested a full tablet report, sending...
I20260618 22:27:27.612571 24566 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:27.613608 23466 ts_manager.cc:194] Re-registered known tserver with Master: c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019)
I20260618 22:27:27.613719 24249 ts_manager.cc:194] Registered new tserver with Master: c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019)
I20260618 22:27:27.614166 24567 heartbeater.cc:511] Master 127.16.243.189:46553 requested a full tablet report, sending...
I20260618 22:27:27.614223 23466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:34209
I20260618 22:27:27.615157 24059 ts_manager.cc:194] Registered new tserver with Master: c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019)
W20260618 22:27:27.715706 24572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:27.715976 24572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:27.716020 24572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:27.720080 24572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:27.720163 24572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:27.720247 24572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:27.725612 24572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:38965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=34221
--tserver_master_addrs=127.16.243.190:43117,127.16.243.189:46553,127.16.243.188:39941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:27.726753 24572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:27.727780 24572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:27.737439 24578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:27.737447 24577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.737524 24572 server_base.cc:1034] running on GCE node
W20260618 22:27:27.737720 24580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:27.738126 24572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:27.738797 24572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:27.739954 24572 hybrid_clock.cc:648] HybridClock initialized: now 1781821647739932 us; error 35 us; skew 500 ppm
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:27.742801 24572 init.cc:377] Logged in from keytab as oryx/127.16.243.131@KRBTEST.COM (short username oryx)
I20260618 22:27:27.743700 24572 webserver.cc:460] Webserver started at http://127.16.243.131:34221/ using document root <none> and password file <none>
I20260618 22:27:27.744237 24572 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:27.744307 24572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:27.748168 24572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:27:27.750146 24587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.751510 24572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:27.751621 24572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
format_stamp: "Formatted at 2026-06-18 22:27:26 on dist-test-slave-jm5g"
server_key: "9349c24220adf7c612cea05fed4f436b"
server_key_iv: "a84187f522af673426e58b3a8bc15906"
server_key_version: "encryptionkey@0"
I20260618 22:27:27.752153 24572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:27.765558 24572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:27.769387 24572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:27.769596 24572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:27.770359 24572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:27.771488 24572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:27.771551 24572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.771633 24572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:27.771668 24572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:27.782907 24572 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:38965
I20260618 22:27:27.782914 24700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:38965 every 8 connection(s)
I20260618 22:27:27.784296 24572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:27.788017 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24572
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.189@KRBTEST.COM
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.188@KRBTEST.COM
I20260618 22:27:27.800437 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.790422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.16.243.188:39941)
0618 22:27:27.790674 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.790686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.790699 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:27.790732 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.790735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.790776 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.790855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.791822 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.792513 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.793688 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.794001 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.794079 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.795903 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.795907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.795910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.795937 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.797972 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.798430 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.798433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.798435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.798797 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.799085 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.799089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.799090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.799307 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.799419 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.800133 (+ 714us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.800251 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":62,"threads_started":1}
I20260618 22:27:27.802580 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.786897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:39779 (local address 127.16.243.188:39941)
0618 22:27:27.787022 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.787028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.787585 (+ 557us) server_negotiation.cc:407] Connection header received
0618 22:27:27.789723 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.789726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.789780 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.789850 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.791268 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.792068 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.793928 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.794215 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.794281 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.799018 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.799022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.799025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.799049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.800503 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.801185 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.801188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.801189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.801269 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.801556 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.801559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.801560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.801754 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.801820 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.802280 (+ 460us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.802390 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:27:27.802591 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.786172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:45205 (local address 127.16.243.189:46553)
0618 22:27:27.786291 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.786294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.788010 (+ 1716us) server_negotiation.cc:407] Connection header received
0618 22:27:27.789225 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.789229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.789286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.789371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.791265 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.792086 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.793966 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.794324 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.794401 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.798215 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.798219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.798223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.798255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.800109 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.800738 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.800742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.800744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.800832 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.801222 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.801226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.801228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.801441 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.801516 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.802277 (+ 761us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.802402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:27:27.804591 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.786801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.16.243.131:45205)
0618 22:27:27.787923 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.787936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.788981 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.789701 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.789713 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.790285 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.791097 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.791116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.792249 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.792252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.793823 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.793834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.794592 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.794595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.794715 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.795336 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.795358 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.797952 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.800236 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.800243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.800262 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.800605 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.800934 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.800937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.800939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.801117 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.801581 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.801588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.801986 (+ 398us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.802799 (+ 813us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.803175 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":189,"spinlock_wait_cycles":28160,"thread_start_us":170,"threads_started":1}
I20260618 22:27:27.804625 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.786802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.16.243.131:39779)
0618 22:27:27.787415 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.787478 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.789349 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.790355 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.790359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.790544 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.791098 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.791116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.792234 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.792237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.793817 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.793826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.794459 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.794463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.794681 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.795448 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.795462 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.798890 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.800844 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.800847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.800849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.801067 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.801373 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.801375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.801377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.801460 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.801886 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.801889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.801984 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.803116 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.803272 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":145,"spinlock_wait_cycles":13952,"thread_start_us":649,"threads_started":1}
I20260618 22:27:27.805859 24703 heartbeater.cc:347] Connected to a master server at 127.16.243.189:46553
I20260618 22:27:27.805902 24701 heartbeater.cc:347] Connected to a master server at 127.16.243.188:39941
I20260618 22:27:27.806102 24703 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.806102 24701 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.806770 24701 heartbeater.cc:511] Master 127.16.243.188:39941 requested a full tablet report, sending...
I20260618 22:27:27.806770 24703 heartbeater.cc:511] Master 127.16.243.189:46553 requested a full tablet report, sending...
I20260618 22:27:27.807986 24249 ts_manager.cc:194] Registered new tserver with Master: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:27.808555 24059 ts_manager.cc:194] Registered new tserver with Master: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:27.810978 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.796612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.16.243.190:43117)
0618 22:27:27.801658 (+ 5046us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.801665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.801680 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:27.801718 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.801721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.801772 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.801851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.802819 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.803559 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.804680 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.805029 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.805121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.806872 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.806876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.806879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.806907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.808725 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.809229 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.809232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.809234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.809326 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.809589 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.809592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.809594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.809811 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.809931 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.810673 (+ 742us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.810807 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4910,"thread_start_us":80,"threads_started":1}
Jun 18 22:27:27 dist-test-slave-jm5g krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821647, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:27.815855 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.806649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.243.189:46553)
0618 22:27:27.806762 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.806766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.807227 (+ 461us) server_negotiation.cc:407] Connection header received
0618 22:27:27.807266 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.807269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.807315 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.807383 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.808082 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.808843 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.809907 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.810235 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.810326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.811914 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.811918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.811921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.811945 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.813579 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.814067 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.814071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.814073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.814159 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.814653 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.814656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.814658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.814883 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.815007 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.815566 (+ 559us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.815695 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:27.814153 23466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:27:27.816820 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.788297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:44699 (local address 127.16.243.190:43117)
0618 22:27:27.788423 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.788427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.789706 (+ 1279us) server_negotiation.cc:407] Connection header received
0618 22:27:27.789869 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.789872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.789918 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.789966 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:27.797915 (+ 7949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.798717 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.810054 (+ 11337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.810360 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.810429 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.813578 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:27.813582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:27.813585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:27.813609 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:27.815100 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.815670 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.815674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.815676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.815756 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:27.816013 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:27.816016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:27.816018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:27.816198 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:27.816265 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.816558 (+ 293us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.816675 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:27:27.817305 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.787963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:44699)
0618 22:27:27.789624 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.789637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.789768 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.790098 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.790100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.795715 (+ 5615us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:27.795823 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.795831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.799327 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.799329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.804335 (+ 5006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.804346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.810727 (+ 6381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.810731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.810830 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.811893 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:27.811908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:27.813414 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:27.815226 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.815229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.815231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.815459 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.815852 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:27.815854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:27.815856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:27.815930 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:27.816315 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:27.816317 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:27.816403 (+ 86us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.816986 (+ 583us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.817144 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":83,"threads_started":1}
W20260618 22:27:27.817740 23466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:27.817946 24702 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43117
I20260618 22:27:27.818041 24702 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:27.818222 24702 heartbeater.cc:511] Master 127.16.243.190:43117 requested a full tablet report, sending...
I20260618 22:27:27.818907 23465 ts_manager.cc:194] Re-registered known tserver with Master: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:27.819620 23465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:44699
I20260618 22:27:27.836203 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.830724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:38965 (local address 127.0.0.1:52464)
0618 22:27:27.831019 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.831034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.831130 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.831710 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.831714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.831734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.831847 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.831853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.833407 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.833411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.834664 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.834683 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.835786 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.835790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.835885 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.835913 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.835962 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.836042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":139,"threads_started":1}
I20260618 22:27:27.836339 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.831303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34405 (local address 127.0.0.1:53874)
0618 22:27:27.831531 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.831544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.831624 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.832340 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.832343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.832361 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.832443 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.832449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.833864 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.833868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.835024 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.835033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.836027 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.836030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.836108 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.836129 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.836170 (+ 41us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.836240 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
I20260618 22:27:27.836980 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.830869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.16.243.131:38965)
0618 22:27:27.831164 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.831177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.831201 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:27:27.831282 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.831289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.831534 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.832012 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.832141 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.833279 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.834820 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.835629 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.836520 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.836672 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.836737 (+ 65us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.836834 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
I20260618 22:27:27.838119 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.831469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.16.243.129:34405)
0618 22:27:27.831874 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.831886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.831907 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:27:27.831981 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.831985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.832191 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.832587 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.832704 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.833746 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.835155 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.835913 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.836912 (+ 999us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.837837 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.837887 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.837983 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":117,"threads_started":1}
I20260618 22:27:27.838353 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.831901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42019 (local address 127.0.0.1:52480)
0618 22:27:27.832111 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.832123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.832212 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.833162 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.833165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.833183 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.833276 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.833282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.834722 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.834725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.835972 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.835983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.837983 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.837987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.838081 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.838107 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.838148 (+ 41us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.838215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260618 22:27:27.839103 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.832438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.16.243.130:42019)
0618 22:27:27.832718 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.832729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.832750 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:27:27.832818 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.832823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.833010 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.833195 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.833401 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.834575 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.836995 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.837847 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.838698 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.838830 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.838876 (+ 46us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.838966 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260618 22:27:27.840157 24635 tablet_service.cc:1467] Processing CreateTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca (DEFAULT_TABLE table=test-table [id=392e7b2748504e26b5aa9de3c9838114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:27.841064 24362 tablet_service.cc:1467] Processing CreateTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca (DEFAULT_TABLE table=test-table [id=392e7b2748504e26b5aa9de3c9838114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:27.841607 24635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85cbdd0a4c64051aa5cae8a889026ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:27.842092 24362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85cbdd0a4c64051aa5cae8a889026ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:27.841755 24499 tablet_service.cc:1467] Processing CreateTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca (DEFAULT_TABLE table=test-table [id=392e7b2748504e26b5aa9de3c9838114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:27.843562 24499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85cbdd0a4c64051aa5cae8a889026ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:27.850121 24727 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Bootstrap starting.
I20260618 22:27:27.852542 24727 tablet_bootstrap.cc:654] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:27.853199 24729 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Bootstrap starting.
I20260618 22:27:27.853454 24727 log.cc:824] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:27.854180 24728 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Bootstrap starting.
I20260618 22:27:27.855635 24729 tablet_bootstrap.cc:654] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:27.855711 24727 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: No bootstrap required, opened a new log
I20260618 22:27:27.855897 24727 ts_tablet_manager.cc:1397] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:27.856251 24728 tablet_bootstrap.cc:654] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:27.856485 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.851831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:58124)
0618 22:27:27.852282 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.852295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.852379 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.852719 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.852722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.852739 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.852832 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.852838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.853833 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.853836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.854972 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.854985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.856074 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.856078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.856167 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.856190 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.856263 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.856350 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378}
I20260618 22:27:27.856900 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.852083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.16.243.188:39941)
0618 22:27:27.852217 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.852221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.852349 (+ 128us) server_negotiation.cc:407] Connection header received
0618 22:27:27.852460 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.852463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.852522 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.852599 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.852959 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.853708 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.855129 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.855748 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.855840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.855979 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.856706 (+ 727us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.856788 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:27:27.857053 24728 log.cc:824] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:27.857753 24259 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260618 22:27:27.859987 24728 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: No bootstrap required, opened a new log
I20260618 22:27:27.860185 24728 ts_tablet_manager.cc:1397] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:27:27.860551 24727 raft_consensus.cc:348] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.860831 24727 raft_consensus.cc:374] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:27.860889 24727 raft_consensus.cc:729] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7, State: Initialized, Role: FOLLOWER
I20260618 22:27:27.861438 24727 consensus_queue.cc:260] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.862537 24727 ts_tablet_manager.cc:1428] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260618 22:27:27.862654 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b1e3ffddce4b4da797bdae9199ca49. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:27.862860 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: LEARNER
I20260618 22:27:27.863543 24702 heartbeater.cc:503] Master 127.16.243.190:43117 was elected leader, sending a full tablet report...
I20260618 22:27:27.863552 24729 log.cc:824] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:27.865243 24728 raft_consensus.cc:348] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.865506 24728 raft_consensus.cc:374] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:27.865549 24728 raft_consensus.cc:729] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c852ec510b6d419ab1506b753219c924, State: Initialized, Role: FOLLOWER
I20260618 22:27:27.866044 24728 consensus_queue.cc:260] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.867131 24728 ts_tablet_manager.cc:1428] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260618 22:27:27.867318 24429 heartbeater.cc:503] Master 127.16.243.190:43117 was elected leader, sending a full tablet report...
I20260618 22:27:27.871879 24729 tablet_bootstrap.cc:492] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: No bootstrap required, opened a new log
I20260618 22:27:27.872085 24729 ts_tablet_manager.cc:1397] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260618 22:27:27.876362 24729 raft_consensus.cc:348] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.876611 24729 raft_consensus.cc:374] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:27.876664 24729 raft_consensus.cc:729] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9930d5de0b0402f9f1860797eb640e3, State: Initialized, Role: FOLLOWER
I20260618 22:27:27.877079 24729 consensus_queue.cc:260] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.877770 24566 heartbeater.cc:503] Master 127.16.243.190:43117 was elected leader, sending a full tablet report...
I20260618 22:27:27.878052 24729 ts_tablet_manager.cc:1428] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260618 22:27:27.897511 24431 tablet.cc:2369] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:27.989910 24740 raft_consensus.cc:482] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:27.990135 24740 raft_consensus.cc:504] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.991325 24740 leader_election.cc:288] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965), c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
I20260618 22:27:27.995057 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.991737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:48297 (local address 127.16.243.129:34405)
0618 22:27:27.991847 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.991850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.991964 (+ 114us) server_negotiation.cc:407] Connection header received
0618 22:27:27.992086 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.992088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.992130 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.992222 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.992458 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.992944 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994033 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.994437 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994611 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.994670 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.994870 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.994944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:27.995059 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.991721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:44271 (local address 127.16.243.131:38965)
0618 22:27:27.991853 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:27.991858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:27.991875 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:27:27.991989 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:27.991992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:27.992083 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:27.992140 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:27.992430 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.992984 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994044 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.994440 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994572 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.994628 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:27.994861 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 22:27:27.994933 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:27:27.995057 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.991548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:38965 (local address 127.16.243.130:44271)
0618 22:27:27.991758 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.991774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.991872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.992173 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.992176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.992191 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.992321 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.992328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.993105 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.993108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.993917 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.993926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.994577 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.994734 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.994764 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.994827 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.994903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":33}
I20260618 22:27:27.995061 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:27.991633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34405 (local address 127.16.243.130:48297)
0618 22:27:27.991880 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:27.991894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:27.991988 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:27.992247 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:27.992249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:27.992263 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:27.992354 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.992358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.993040 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.993044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.993908 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:27.993917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:27.994582 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:27.994585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:27.994761 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:27.994777 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:27:27.994847 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:27:27.994920 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":107}
I20260618 22:27:27.995721 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85cbdd0a4c64051aa5cae8a889026ca" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" is_pre_election: true
I20260618 22:27:27.995879 24382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85cbdd0a4c64051aa5cae8a889026ca" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c852ec510b6d419ab1506b753219c924" is_pre_election: true
I20260618 22:27:27.995999 24655 raft_consensus.cc:2454] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 0.
I20260618 22:27:27.996181 24382 raft_consensus.cc:2454] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 0.
I20260618 22:27:27.996443 24454 leader_election.cc:302] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7, c9930d5de0b0402f9f1860797eb640e3; no voters:
I20260618 22:27:27.996742 24740 raft_consensus.cc:2781] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:27.996820 24740 raft_consensus.cc:482] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:27.996872 24740 raft_consensus.cc:3037] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:27.998368 24740 raft_consensus.cc:504] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:27.998762 24740 leader_election.cc:288] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965), c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405)
I20260618 22:27:27.999161 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85cbdd0a4c64051aa5cae8a889026ca" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
I20260618 22:27:27.999178 24382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85cbdd0a4c64051aa5cae8a889026ca" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c852ec510b6d419ab1506b753219c924"
I20260618 22:27:27.999286 24382 raft_consensus.cc:3037] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:27.999292 24655 raft_consensus.cc:3037] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:28.000270 24382 raft_consensus.cc:2454] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 1.
I20260618 22:27:28.000278 24655 raft_consensus.cc:2454] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 1.
I20260618 22:27:28.000541 24453 leader_election.cc:302] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c852ec510b6d419ab1506b753219c924, c9930d5de0b0402f9f1860797eb640e3; no voters:
I20260618 22:27:28.000722 24740 raft_consensus.cc:2781] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:28.000959 24740 raft_consensus.cc:686] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 1 LEADER]: Becoming Leader. State: Replica: c9930d5de0b0402f9f1860797eb640e3, State: Running, Role: LEADER
I20260618 22:27:28.001376 24740 consensus_queue.cc:237] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:28.004998 23465 catalog_manager.cc:5495] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "c9930d5de0b0402f9f1860797eb640e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:28.006628 24026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: NON_VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: true } } }
I20260618 22:27:28.006770 24026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:28.010080 24743 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: attempting to promote NON_VOTER 3abdcce626814229bf33fd386366766c to VOTER
I20260618 22:27:28.010444 24743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:28.010948 24026 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: attempt to promote peer 3abdcce626814229bf33fd386366766c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260618 22:27:28.011507 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 22:27:28.011513 24259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 LEARNER]: Refusing update from remote peer 71b1e3ffddce4b4da797bdae9199ca49: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 22:27:28.011817 24743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 22:27:28.012014 24717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 22:27:28.013577 24745 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3abdcce626814229bf33fd386366766c (127.16.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.013978 24077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3abdcce626814229bf33fd386366766c (127.16.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.014104 24259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3abdcce626814229bf33fd386366766c (127.16.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.015108 24745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.015218 24745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:28.015388 24745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.015470 24745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:28.015453 24717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.015565 24717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:28.015841 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.015949 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:28.015899 24746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71b1e3ffddce4b4da797bdae9199ca49" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:28.016027 24746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:28.033553 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:28.030237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.16.243.130:42019)
0618 22:27:28.030369 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:28.030374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:28.030388 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:28.030469 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:28.030472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:28.030524 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:28.030635 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:28.030849 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.031349 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.032153 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.032377 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.032469 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:28.032741 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:28.033157 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:28.033274 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:28.033337 (+ 63us) server_negotiation.cc:299] Negotiation successful
0618 22:27:28.033396 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 22:27:28.036064 24704 tablet.cc:2369] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:28.039291 24655 raft_consensus.cc:1261] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Refusing update from remote peer c9930d5de0b0402f9f1860797eb640e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:28.039292 24382 raft_consensus.cc:1261] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Refusing update from remote peer c9930d5de0b0402f9f1860797eb640e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:28.039959 24741 consensus_queue.cc:1035] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:28.040126 24740 consensus_queue.cc:1035] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:28.049074 24750 mvcc.cc:205] Tried to move back new op lower bound from 7298341470362198016 to 7298341470216429568. Current Snapshot: MvccSnapshot[applied={T|T < 7298341470362198016}]
I20260618 22:27:28.050999 24749 mvcc.cc:205] Tried to move back new op lower bound from 7298341470362198016 to 7298341470216429568. Current Snapshot: MvccSnapshot[applied={T|T < 7298341470362198016}]
I20260618 22:27:28.060359 23466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:28.060559 23466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:28.065349 23466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71b1e3ffddce4b4da797bdae9199ca49: Sending DeleteTablet for 3 replicas of tablet f85cbdd0a4c64051aa5cae8a889026ca
I20260618 22:27:28.066179 24635 tablet_service.cc:1514] Processing DeleteTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:28 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:52464
I20260618 22:27:28.066447 24499 tablet_service.cc:1514] Processing DeleteTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:28 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:52480
I20260618 22:27:28.066704 24362 tablet_service.cc:1514] Processing DeleteTablet for tablet f85cbdd0a4c64051aa5cae8a889026ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:28 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:53874
I20260618 22:27:28.066998 24759 tablet_replica.cc:330] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: stopping tablet replica
I20260618 22:27:28.067384 24759 raft_consensus.cc:2229] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:28.067705 24758 tablet_replica.cc:330] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: stopping tablet replica
I20260618 22:27:28.067917 24759 raft_consensus.cc:2258] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:28.068022 24758 raft_consensus.cc:2229] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:28.068372 24758 raft_consensus.cc:2258] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:28.069607 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23434
I20260618 22:27:28.070376 24759 ts_tablet_manager.cc:1905] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:28.070797 24758 ts_tablet_manager.cc:1905] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:28.073369 24759 ts_tablet_manager.cc:1918] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:28.073464 24758 ts_tablet_manager.cc:1918] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:28.073559 24758 log.cc:1196] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/f85cbdd0a4c64051aa5cae8a889026ca
I20260618 22:27:28.073571 24759 log.cc:1196] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/f85cbdd0a4c64051aa5cae8a889026ca
I20260618 22:27:28.073869 24758 ts_tablet_manager.cc:1939] T f85cbdd0a4c64051aa5cae8a889026ca P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting consensus metadata
I20260618 22:27:28.074043 24759 ts_tablet_manager.cc:1939] T f85cbdd0a4c64051aa5cae8a889026ca P c9930d5de0b0402f9f1860797eb640e3: Deleting consensus metadata
W20260618 22:27:28.076296 24454 connection.cc:535] server connection from 127.0.0.1:52480 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 22:27:28.076612 24589 connection.cc:535] server connection from 127.0.0.1:52464 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:27:28.077793 24760 tablet_replica.cc:330] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: stopping tablet replica
I20260618 22:27:28.078162 24760 raft_consensus.cc:2229] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:28.078521 24760 raft_consensus.cc:2258] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:28.080260 24760 ts_tablet_manager.cc:1905] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:28.083299 24760 ts_tablet_manager.cc:1918] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:28.083412 24760 log.cc:1196] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/f85cbdd0a4c64051aa5cae8a889026ca
I20260618 22:27:28.083787 24760 ts_tablet_manager.cc:1939] T f85cbdd0a4c64051aa5cae8a889026ca P c852ec510b6d419ab1506b753219c924: Deleting consensus metadata
W20260618 22:27:28.084970 24316 connection.cc:441] server connection from 127.0.0.1:53874 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53874 (request call id 1) could send its response
I20260618 22:27:28.088864 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:28.068679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.16.243.189:46553)
0618 22:27:28.068903 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:28.068908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:28.077517 (+ 8609us) server_negotiation.cc:407] Connection header received
0618 22:27:28.077630 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:28.077633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:28.077689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:28.077765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:28.078995 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.079824 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.081118 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.081475 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.081544 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:28.083664 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:28.083668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:28.083671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:28.083698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:28.085494 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.087538 (+ 2044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.087542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.087544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.087636 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.087920 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.087924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.087926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.088131 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:28.088253 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:28.088569 (+ 316us) server_negotiation.cc:299] Negotiation successful
0618 22:27:28.088692 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260618 22:27:28.092133 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:28.082228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.16.243.188:39941)
0618 22:27:28.082361 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:28.082366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:28.082600 (+ 234us) server_negotiation.cc:407] Connection header received
0618 22:27:28.082776 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:28.082779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:28.082829 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:28.082878 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:28.084574 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.085279 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.086134 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.086382 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.086472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:28.087911 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:28.087915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:28.087918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:28.087947 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:28.089542 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.090764 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.090769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.090770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.090866 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.091172 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.091175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.091177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.091384 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:28.091499 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:28.091848 (+ 349us) server_negotiation.cc:299] Negotiation successful
0618 22:27:28.091972 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 22:27:28.093722 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:28.079771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.16.243.189:46553)
0618 22:27:28.080334 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:28.080345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:28.080923 (+ 578us) server_negotiation.cc:407] Connection header received
0618 22:27:28.080963 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:28.080966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:28.081011 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:28.081084 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:28.083668 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.084424 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.087130 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.087511 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.087612 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:28.089702 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:28.089706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:28.089708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:28.089731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:28.091539 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.091941 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.091944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.091946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.092023 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.092308 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.092312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.092313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.092507 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:28.092620 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:28.093213 (+ 593us) server_negotiation.cc:299] Negotiation successful
0618 22:27:28.093337 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":72,"threads_started":1}
I20260618 22:27:28.097875 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:28.068791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.16.243.188:39941)
0618 22:27:28.068983 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:28.068987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:28.089617 (+ 20630us) server_negotiation.cc:407] Connection header received
0618 22:27:28.089779 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:28.089782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:28.089828 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:28.089872 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:28.090610 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.091379 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.092714 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:28.093042 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:28.093109 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:28.094735 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:28.094738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:28.094740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:28.094761 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:28.096227 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.096596 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.096599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.096601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.096684 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:28.096955 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:28.096958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:28.096960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:28.097181 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:28.097300 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:28.097610 (+ 310us) server_negotiation.cc:299] Negotiation successful
0618 22:27:28.097723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260618 22:27:29.052250 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.051787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:33273)
0618 22:27:29.052028 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.052141 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
W20260618 22:27:29.053018 24702 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.052763 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.052072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:50321)
0618 22:27:29.052589 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.052674 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260618 22:27:29.053678 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.052705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:46079)
0618 22:27:29.053486 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.053585 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":61,"threads_started":1}
W20260618 22:27:29.053700 24566 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
W20260618 22:27:29.053970 24429 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.566062 24772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71b1e3ffddce4b4da797bdae9199ca49)
I20260618 22:27:29.566090 24773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71b1e3ffddce4b4da797bdae9199ca49)
I20260618 22:27:29.566304 24772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:29.566304 24773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:29.567279 24773 leader_election.cc:288] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189:46553)
I20260618 22:27:29.567651 24772 leader_election.cc:288] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 3abdcce626814229bf33fd386366766c (127.16.243.188:39941)
I20260618 22:27:29.568046 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.567457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:39996)
0618 22:27:29.567863 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.567941 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":204,"thread_start_us":58,"threads_started":1}
I20260618 22:27:29.568073 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.567458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:39994)
0618 22:27:29.567935 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.567997 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":77,"threads_started":1}
W20260618 22:27:29.568236 24236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
W20260618 22:27:29.568351 24054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.569166 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.568934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40008)
0618 22:27:29.569051 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.569098 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260618 22:27:29.569164 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.568934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40024)
0618 22:27:29.569051 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.569109 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260618 22:27:29.569298 24054 leader_election.cc:334] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
W20260618 22:27:29.569298 24236 leader_election.cc:334] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.572516 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.567457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:39941 (local address 127.0.0.1:58130)
0618 22:27:29.567825 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.567842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:29.567946 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:29.568652 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:29.568654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:29.568667 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:29.568768 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:29.568774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.569913 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.569917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:29.571250 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:29.571261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.572095 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.572098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:29.572195 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:29.572220 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:27:29.572292 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:27:29.572370 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":63,"threads_started":1}
I20260618 22:27:29.572542 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.567792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:46553 (local address 127.0.0.1:51670)
0618 22:27:29.568197 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.568210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:29.568312 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:29.568758 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:29.568760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:29.568776 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:29.568866 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:29.568873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.569947 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.569950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:29.571250 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:29.571269 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.572095 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.572098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:29.572195 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:29.572220 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:27:29.572302 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:27:29.572370 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":58,"threads_started":1}
I20260618 22:27:29.572558 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.567947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.16.243.188:39941)
0618 22:27:29.568432 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:29.568436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:29.568450 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:29.568494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:29.568497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:29.568539 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:29.568856 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:29.568922 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.569795 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.571365 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.571986 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.572103 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:29.572153 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:29.572365 (+ 212us) server_negotiation.cc:299] Negotiation successful
0618 22:27:29.572435 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":50,"threads_started":1}
I20260618 22:27:29.572546 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.568140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.16.243.189:46553)
0618 22:27:29.568492 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:29.568496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:29.568506 (+ 10us) server_negotiation.cc:407] Connection header received
0618 22:27:29.568541 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:29.568543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:29.568582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:29.568651 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:29.568968 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.569852 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.571371 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:29.571986 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:29.572103 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:29.572153 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:29.572332 (+ 179us) server_negotiation.cc:299] Negotiation successful
0618 22:27:29.572411 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":50,"threads_started":1}
I20260618 22:27:29.572973 24259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13287b61dedb4f3fbe97ed2159a93302" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3abdcce626814229bf33fd386366766c" is_pre_election: true
I20260618 22:27:29.573092 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3abdcce626814229bf33fd386366766c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13287b61dedb4f3fbe97ed2159a93302" is_pre_election: true
I20260618 22:27:29.573241 24259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13287b61dedb4f3fbe97ed2159a93302 in term 1.
I20260618 22:27:29.573297 24077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3abdcce626814229bf33fd386366766c in term 1.
I20260618 22:27:29.573606 24236 leader_election.cc:302] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13287b61dedb4f3fbe97ed2159a93302, 3abdcce626814229bf33fd386366766c; no voters: 71b1e3ffddce4b4da797bdae9199ca49
I20260618 22:27:29.573627 24051 leader_election.cc:302] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13287b61dedb4f3fbe97ed2159a93302, 3abdcce626814229bf33fd386366766c; no voters: 71b1e3ffddce4b4da797bdae9199ca49
I20260618 22:27:29.573807 24772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:27:29.573861 24772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71b1e3ffddce4b4da797bdae9199ca49)
I20260618 22:27:29.573899 24772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:27:29.573858 24773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:27:29.573951 24773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71b1e3ffddce4b4da797bdae9199ca49)
I20260618 22:27:29.573993 24773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:27:29.575040 24773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:29.575057 24772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:29.575387 24773 leader_election.cc:288] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 election: Requested vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189:46553)
I20260618 22:27:29.575407 24772 leader_election.cc:288] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 election: Requested vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 3abdcce626814229bf33fd386366766c (127.16.243.188:39941)
I20260618 22:27:29.575752 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.575529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40036)
0618 22:27:29.575643 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.575691 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:27:29.575956 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.575742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40040)
0618 22:27:29.575847 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.575896 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:27:29.576015 24259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13287b61dedb4f3fbe97ed2159a93302" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3abdcce626814229bf33fd386366766c"
I20260618 22:27:29.576066 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3abdcce626814229bf33fd386366766c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13287b61dedb4f3fbe97ed2159a93302"
I20260618 22:27:29.576162 24259 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 13287b61dedb4f3fbe97ed2159a93302 in current term 2: Already voted for candidate 3abdcce626814229bf33fd386366766c in this term.
I20260618 22:27:29.576196 24077 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3abdcce626814229bf33fd386366766c in current term 2: Already voted for candidate 13287b61dedb4f3fbe97ed2159a93302 in this term.
I20260618 22:27:29.576640 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.576450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40048)
0618 22:27:29.576548 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.576592 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 22:27:29.576736 24236 leader_election.cc:334] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.576802 24236 leader_election.cc:302] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3abdcce626814229bf33fd386366766c; no voters: 13287b61dedb4f3fbe97ed2159a93302, 71b1e3ffddce4b4da797bdae9199ca49
I20260618 22:27:29.576948 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:29.576712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40064)
0618 22:27:29.576840 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:29.576887 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260618 22:27:29.576929 24773 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260618 22:27:29.577042 24054 leader_election.cc:334] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:29.577112 24054 leader_election.cc:302] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13287b61dedb4f3fbe97ed2159a93302; no voters: 3abdcce626814229bf33fd386366766c, 71b1e3ffddce4b4da797bdae9199ca49
I20260618 22:27:29.577226 24772 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260618 22:27:30.054188 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:30.053596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:58011)
0618 22:27:30.053996 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:30.054074 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260618 22:27:30.054807 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:30.054147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:43889)
0618 22:27:30.054658 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:30.054718 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":75,"threads_started":1}
I20260618 22:27:30.055153 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:30.054729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:39789)
0618 22:27:30.055007 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:30.055064 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260618 22:27:31.055430 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.055007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:43515)
0618 22:27:31.055251 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.055321 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260618 22:27:31.056141 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.055664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:60623)
0618 22:27:31.055995 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.056052 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":63,"threads_started":1}
I20260618 22:27:31.056655 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.055664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:52335)
0618 22:27:31.056533 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.056585 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":70,"threads_started":1}
I20260618 22:27:31.449483 24788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:31.449621 24788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:31.449986 24788 leader_election.cc:288] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189:46553)
I20260618 22:27:31.450542 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.450173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40298)
0618 22:27:31.450386 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.450453 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260618 22:27:31.450757 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3abdcce626814229bf33fd386366766c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13287b61dedb4f3fbe97ed2159a93302" is_pre_election: true
I20260618 22:27:31.450906 24077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3abdcce626814229bf33fd386366766c in term 2.
I20260618 22:27:31.451253 24236 leader_election.cc:302] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13287b61dedb4f3fbe97ed2159a93302, 3abdcce626814229bf33fd386366766c; no voters:
I20260618 22:27:31.451341 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.451121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40302)
0618 22:27:31.451235 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.451287 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 22:27:31.451409 24788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260618 22:27:31.451434 24236 leader_election.cc:334] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:31.451467 24788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:31.451505 24788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:27:31.452473 24788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:31.452762 24788 leader_election.cc:288] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 election: Requested vote from peers 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117), 13287b61dedb4f3fbe97ed2159a93302 (127.16.243.189:46553)
I20260618 22:27:31.453081 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.452882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40314)
0618 22:27:31.452966 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.453018 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260618 22:27:31.453280 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3abdcce626814229bf33fd386366766c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13287b61dedb4f3fbe97ed2159a93302"
I20260618 22:27:31.453398 24077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:27:31.454312 24077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3abdcce626814229bf33fd386366766c in term 3.
I20260618 22:27:31.454608 24236 leader_election.cc:302] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13287b61dedb4f3fbe97ed2159a93302, 3abdcce626814229bf33fd386366766c; no voters:
I20260618 22:27:31.454752 24788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:27:31.454937 24788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [term 3 LEADER]: Becoming Leader. State: Replica: 3abdcce626814229bf33fd386366766c, State: Running, Role: LEADER
I20260618 22:27:31.455089 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.454861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40320)
0618 22:27:31.454963 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.455015 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:27:31.455258 24236 leader_election.cc:334] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
I20260618 22:27:31.455262 24788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } }
I20260618 22:27:31.456420 24791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abdcce626814229bf33fd386366766c. Latest consensus state: current_term: 3 leader_uuid: "3abdcce626814229bf33fd386366766c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:31.456782 24791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:31.457052 24793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:31.459270 24793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:31.459638 24793 catalog_manager.cc:1259] Loaded cluster ID: 7915d150cb624984aa5be62bb4021d82
I20260618 22:27:31.459681 24793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:31.460207 24793 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:31.460578 24793 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Loaded TSK: 0
I20260618 22:27:31.461234 24793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:31.548272 24249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:31.551437 24249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:31.554497 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.554232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40338)
0618 22:27:31.554377 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.554430 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:27:31.554490 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [term 3 FOLLOWER]: Refusing update from remote peer 3abdcce626814229bf33fd386366766c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260618 22:27:31.554950 24791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260618 22:27:31.555224 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.555016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.0.0.1:40344)
0618 22:27:31.555115 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.555168 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:27:31.555426 24236 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c -> Peer 71b1e3ffddce4b4da797bdae9199ca49 (127.16.243.190:43117): Couldn't send request to peer 71b1e3ffddce4b4da797bdae9199ca49. Status: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 22:27:31.556339 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abdcce626814229bf33fd386366766c. Latest consensus state: current_term: 3 leader_uuid: "3abdcce626814229bf33fd386366766c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:31.556452 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:31.557478 24788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3abdcce626814229bf33fd386366766c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:31.557596 24788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:31.558164 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3abdcce626814229bf33fd386366766c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "71b1e3ffddce4b4da797bdae9199ca49" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43117 } } peers { permanent_uuid: "13287b61dedb4f3fbe97ed2159a93302" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 46553 } attrs { promote: false } } peers { permanent_uuid: "3abdcce626814229bf33fd386366766c" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 39941 } attrs { promote: false } } }
I20260618 22:27:31.558310 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13287b61dedb4f3fbe97ed2159a93302 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:31.567634 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.563617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:38965 (local address 127.0.0.1:52478)
0618 22:27:31.563845 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.563857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:31.563947 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:31.564389 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:31.564392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:31.564409 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:31.564488 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.564493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.565270 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.565274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.566251 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.566260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.567153 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.567156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.567313 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.567335 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:31.567403 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:27:31.567469 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":82,"threads_started":1}
I20260618 22:27:31.567649 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.563947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.16.243.131:38965)
0618 22:27:31.564163 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:31.564167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:31.564180 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:31.564232 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:31.564235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:31.564280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:31.564365 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:31.564592 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.565146 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.566374 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.567027 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.567234 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.567283 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:31.567428 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 22:27:31.567503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260618 22:27:31.567843 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.563163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34405 (local address 127.0.0.1:53888)
0618 22:27:31.563306 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.563318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:31.563406 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:31.563866 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:31.563869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:31.563886 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:31.564006 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.564014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.565005 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.565008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.566142 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.566152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.567070 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.567074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.567200 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.567246 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 22:27:31.567325 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:27:31.567428 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":35,"spinlock_wait_cycles":1920}
I20260618 22:27:31.568204 24635 tablet_service.cc:1467] Processing CreateTablet for tablet ded4069601024b428421f87c3401f809 (DEFAULT_TABLE table=test-table [id=5d76bc2a6fea4896a34844a85eb72b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:31.568250 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.563278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.16.243.129:34405)
0618 22:27:31.563600 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:31.563604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:31.563617 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:31.563664 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:31.563666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:31.563713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:31.564114 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:31.564201 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.564911 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.566265 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.566961 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.567077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.567130 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:31.568041 (+ 911us) server_negotiation.cc:299] Negotiation successful
0618 22:27:31.568124 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":84,"threads_started":1}
I20260618 22:27:31.568472 24635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded4069601024b428421f87c3401f809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:31.568694 24362 tablet_service.cc:1467] Processing CreateTablet for tablet ded4069601024b428421f87c3401f809 (DEFAULT_TABLE table=test-table [id=5d76bc2a6fea4896a34844a85eb72b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:31.568921 24362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded4069601024b428421f87c3401f809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:31.569236 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.564247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:42019 (local address 127.0.0.1:52504)
0618 22:27:31.565106 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.565118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:31.565202 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:31.565423 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:31.565426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:31.565441 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:31.565557 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.565563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.566516 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.566520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.568114 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:31.568123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.568870 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.568874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:31.568968 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.568989 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:27:31.569052 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:31.569116 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260618 22:27:31.569257 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.564565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.16.243.130:42019)
0618 22:27:31.564795 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:31.564799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:31.565194 (+ 395us) server_negotiation.cc:407] Connection header received
0618 22:27:31.565279 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:31.565282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:31.565331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:31.565408 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:31.565676 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.566409 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.568237 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.568755 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.568870 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.568918 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:31.569072 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 22:27:31.569142 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260618 22:27:31.569685 24499 tablet_service.cc:1467] Processing CreateTablet for tablet ded4069601024b428421f87c3401f809 (DEFAULT_TABLE table=test-table [id=5d76bc2a6fea4896a34844a85eb72b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:31.569885 24499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded4069601024b428421f87c3401f809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:31.571053 24805 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Bootstrap starting.
I20260618 22:27:31.571516 24806 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924: Bootstrap starting.
I20260618 22:27:31.572183 24805 tablet_bootstrap.cc:654] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:31.572412 24806 tablet_bootstrap.cc:654] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:31.572690 24807 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Bootstrap starting.
I20260618 22:27:31.573498 24805 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: No bootstrap required, opened a new log
I20260618 22:27:31.573577 24806 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924: No bootstrap required, opened a new log
I20260618 22:27:31.573601 24807 tablet_bootstrap.cc:654] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:31.573596 24805 ts_tablet_manager.cc:1397] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:31.573702 24806 ts_tablet_manager.cc:1397] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:31.574028 24805 raft_consensus.cc:348] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.574194 24805 raft_consensus.cc:374] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:31.574111 24806 raft_consensus.cc:348] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.574235 24805 raft_consensus.cc:729] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7, State: Initialized, Role: FOLLOWER
I20260618 22:27:31.574244 24806 raft_consensus.cc:374] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:31.574275 24806 raft_consensus.cc:729] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c852ec510b6d419ab1506b753219c924, State: Initialized, Role: FOLLOWER
I20260618 22:27:31.574362 24805 consensus_queue.cc:260] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.574409 24806 consensus_queue.cc:260] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.574729 24806 ts_tablet_manager.cc:1428] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:27:31.574731 24805 ts_tablet_manager.cc:1428] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:31.575213 24807 tablet_bootstrap.cc:492] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: No bootstrap required, opened a new log
I20260618 22:27:31.575346 24807 ts_tablet_manager.cc:1397] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:31.575604 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.575189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:48465)
0618 22:27:31.575439 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.575502 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260618 22:27:31.576184 24807 raft_consensus.cc:348] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.576292 24807 raft_consensus.cc:374] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:31.576321 24807 raft_consensus.cc:729] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9930d5de0b0402f9f1860797eb640e3, State: Initialized, Role: FOLLOWER
I20260618 22:27:31.576733 24807 consensus_queue.cc:260] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.577054 24807 ts_tablet_manager.cc:1428] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:31.578359 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.575626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:34799)
0618 22:27:31.578188 (+ 2562us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.578250 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2492,"thread_start_us":86,"threads_started":1}
I20260618 22:27:31.578366 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.577983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:53591)
0618 22:27:31.578207 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.578264 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260618 22:27:31.578636 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.577544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:33915)
0618 22:27:31.578495 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.578542 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":887}
I20260618 22:27:31.578662 24428 heartbeater.cc:503] Master 127.16.243.188:39941 was elected leader, sending a full tablet report...
I20260618 22:27:31.578840 24701 heartbeater.cc:503] Master 127.16.243.188:39941 was elected leader, sending a full tablet report...
I20260618 22:27:31.579217 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.579026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:38045)
0618 22:27:31.579119 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.579163 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 22:27:31.580981 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.580474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:43647)
0618 22:27:31.580863 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.580915 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260618 22:27:31.581434 24565 heartbeater.cc:503] Master 127.16.243.188:39941 was elected leader, sending a full tablet report...
W20260618 22:27:31.595094 24568 tablet.cc:2369] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:31.776396 24815 raft_consensus.cc:482] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:31.776544 24815 raft_consensus.cc:504] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.776983 24815 leader_election.cc:288] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405), 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:31.777490 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded4069601024b428421f87c3401f809" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" is_pre_election: true
I20260618 22:27:31.777490 24382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded4069601024b428421f87c3401f809" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c852ec510b6d419ab1506b753219c924" is_pre_election: true
I20260618 22:27:31.777649 24655 raft_consensus.cc:2454] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 0.
I20260618 22:27:31.777657 24382 raft_consensus.cc:2454] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 0.
I20260618 22:27:31.777967 24453 leader_election.cc:302] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c852ec510b6d419ab1506b753219c924, c9930d5de0b0402f9f1860797eb640e3; no voters:
I20260618 22:27:31.778144 24815 raft_consensus.cc:2781] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:31.778221 24815 raft_consensus.cc:482] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:31.778266 24815 raft_consensus.cc:3037] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:31.779354 24815 raft_consensus.cc:504] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.779682 24815 leader_election.cc:288] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 election: Requested vote from peers c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405), 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965)
I20260618 22:27:31.780081 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded4069601024b428421f87c3401f809" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7"
I20260618 22:27:31.780071 24382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded4069601024b428421f87c3401f809" candidate_uuid: "c9930d5de0b0402f9f1860797eb640e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c852ec510b6d419ab1506b753219c924"
I20260618 22:27:31.780200 24382 raft_consensus.cc:3037] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:31.780200 24655 raft_consensus.cc:3037] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:31.781268 24655 raft_consensus.cc:2454] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 1.
I20260618 22:27:31.781268 24382 raft_consensus.cc:2454] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9930d5de0b0402f9f1860797eb640e3 in term 1.
I20260618 22:27:31.781543 24454 leader_election.cc:302] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7, c9930d5de0b0402f9f1860797eb640e3; no voters:
I20260618 22:27:31.781709 24815 raft_consensus.cc:2781] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:31.781895 24815 raft_consensus.cc:686] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 1 LEADER]: Becoming Leader. State: Replica: c9930d5de0b0402f9f1860797eb640e3, State: Running, Role: LEADER
I20260618 22:27:31.782023 24815 consensus_queue.cc:237] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } }
I20260618 22:27:31.782984 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.782383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:52995)
0618 22:27:31.782499 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.782591 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 22:27:31.783946 24248 catalog_manager.cc:5495] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "c9930d5de0b0402f9f1860797eb640e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9930d5de0b0402f9f1860797eb640e3" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 42019 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:31.805434 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.802599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.16.243.130:42019)
0618 22:27:31.802738 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:31.802742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:31.802759 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:27:31.802808 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:31.802810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:31.802851 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:31.802910 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:31.803150 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.803652 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.804476 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:31.804696 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:31.804778 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:31.805019 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:31.805089 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:31.805145 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:31.805245 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 22:27:31.805300 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:27:31.807571 24382 raft_consensus.cc:1261] T ded4069601024b428421f87c3401f809 P c852ec510b6d419ab1506b753219c924 [term 1 FOLLOWER]: Refusing update from remote peer c9930d5de0b0402f9f1860797eb640e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:31.807582 24655 raft_consensus.cc:1261] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Refusing update from remote peer c9930d5de0b0402f9f1860797eb640e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:31.807984 24817 consensus_queue.cc:1035] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c852ec510b6d419ab1506b753219c924" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:31.808148 24815 consensus_queue.cc:1035] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe5ff7eb4364c9e8dc8fd2e62bfe2b7" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 38965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:31.809790 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.809548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:51571)
0618 22:27:31.809657 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.809715 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260618 22:27:31.811209 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.810880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:37543)
0618 22:27:31.811063 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.811133 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260618 22:27:31.811791 24821 mvcc.cc:205] Tried to move back new op lower bound from 7298341485799772160 to 7298341485700558848. Current Snapshot: MvccSnapshot[applied={T|T < 7298341485799772160}]
I20260618 22:27:31.813791 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.813452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:46349)
0618 22:27:31.813656 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.813719 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260618 22:27:31.815500 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.815059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.131:46333)
0618 22:27:31.815384 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.815438 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255}
I20260618 22:27:31.816460 24248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:31.816627 24248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:31.816910 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.816656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.130:57337)
0618 22:27:31.816790 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.816844 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260618 22:27:31.817409 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.817101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43117 (local address 127.16.243.129:53061)
0618 22:27:31.817287 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.817337 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260618 22:27:31.820793 24248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3abdcce626814229bf33fd386366766c: Sending DeleteTablet for 3 replicas of tablet ded4069601024b428421f87c3401f809
I20260618 22:27:31.821382 24362 tablet_service.cc:1514] Processing DeleteTablet for tablet ded4069601024b428421f87c3401f809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:31 UTC) from {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:53888
I20260618 22:27:31.821558 24499 tablet_service.cc:1514] Processing DeleteTablet for tablet ded4069601024b428421f87c3401f809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:31 UTC) from {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:52504
I20260618 22:27:31.821707 24635 tablet_service.cc:1514] Processing DeleteTablet for tablet ded4069601024b428421f87c3401f809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:31 UTC) from {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:52478
I20260618 22:27:31.821828 24498 tablet_service.cc:1514] Processing DeleteTablet for tablet ded4069601024b428421f87c3401f809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:31 UTC) from {username='oryx', principal='oryx/127.16.243.188@KRBTEST.COM'} at 127.0.0.1:52504
I20260618 22:27:31.821874 24831 tablet_replica.cc:330] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: stopping tablet replica
I20260618 22:27:31.821990 24831 raft_consensus.cc:2229] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:31.822057 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24211
I20260618 22:27:31.822579 24831 raft_consensus.cc:2258] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:31.823592 24831 ts_tablet_manager.cc:1905] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:31.825655 24831 ts_tablet_manager.cc:1918] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:31.825727 24831 log.cc:1196] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ded4069601024b428421f87c3401f809
I20260618 22:27:31.825954 24831 ts_tablet_manager.cc:1939] T ded4069601024b428421f87c3401f809 P c9930d5de0b0402f9f1860797eb640e3: Deleting consensus metadata
I20260618 22:27:31.826642 24236 catalog_manager.cc:4841] TS c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019): tablet ded4069601024b428421f87c3401f809 (table test-table [id=5d76bc2a6fea4896a34844a85eb72b56]) successfully deleted
W20260618 22:27:31.826972 24236 catalog_manager.cc:4808] TS c9930d5de0b0402f9f1860797eb640e3 (127.16.243.130:42019): delete failed for tablet ded4069601024b428421f87c3401f809 because the tablet was not found. No further retry: Not found: Tablet not found: ded4069601024b428421f87c3401f809
I20260618 22:27:31.828266 24832 tablet_replica.cc:330] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: stopping tablet replica
I20260618 22:27:31.828406 24832 raft_consensus.cc:2229] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:31.828558 24832 raft_consensus.cc:2258] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:31.828663 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:31.828431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34405 (local address 127.0.0.1:53896)
0618 22:27:31.828542 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:31.828589 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 22:27:31.828792 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24436
I20260618 22:27:31.829711 24832 ts_tablet_manager.cc:1905] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:27:31.831044 24234 catalog_manager.cc:4584] TS c852ec510b6d419ab1506b753219c924 (127.16.243.129:34405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ded4069601024b428421f87c3401f809: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34405: connect: Connection refused (error 111)
I20260618 22:27:31.832038 24832 ts_tablet_manager.cc:1918] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:31.832101 24832 log.cc:1196] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/ded4069601024b428421f87c3401f809
I20260618 22:27:31.832319 24832 ts_tablet_manager.cc:1939] T ded4069601024b428421f87c3401f809 P 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7: Deleting consensus metadata
I20260618 22:27:31.832948 24235 catalog_manager.cc:4841] TS 3fe5ff7eb4364c9e8dc8fd2e62bfe2b7 (127.16.243.131:38965): tablet ded4069601024b428421f87c3401f809 (table test-table [id=5d76bc2a6fea4896a34844a85eb72b56]) successfully deleted
I20260618 22:27:31.835295 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24572
I20260618 22:27:31.840790 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 23930
I20260618 22:27:31.845244 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24039
2026-06-18T22:27:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7200 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:31 dist-test-slave-jm5g krb5kdc[24835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:31 dist-test-slave-jm5g krb5kdc[24835](info): set up 2 sockets
Jun 18 22:27:31 dist-test-slave-jm5g krb5kdc[24835](info): commencing operation
krb5kdc: starting...
W20260618 22:27:32.802896 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:32 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab.
Entry for principal oryx/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab.
I20260618 22:27:32.921036 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35921
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab
--principal=oryx/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:33.030056 24848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:33.030346 24848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:33.030386 24848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:33.035276 24848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:33.035392 24848 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:27:33.035413 24848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:33.035429 24848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:33.035445 24848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:33.035460 24848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:33.041132 24848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:35921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.190.keytab
--principal=oryx/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:33.042282 24848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:33.043360 24848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:33.051684 24854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:33.051667 24853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.051915 24848 server_base.cc:1034] running on GCE node
W20260618 22:27:33.051667 24856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.052770 24848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:33.053957 24848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:33.055137 24848 hybrid_clock.cc:648] HybridClock initialized: now 1781821653055116 us; error 38 us; skew 500 ppm
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:33.057849 24848 init.cc:377] Logged in from keytab as oryx/127.16.243.190@KRBTEST.COM (short username oryx)
I20260618 22:27:33.058789 24848 webserver.cc:460] Webserver started at http://127.16.243.190:37059/ using document root <none> and password file <none>
I20260618 22:27:33.059335 24848 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:33.059401 24848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:33.059600 24848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:33.061694 24848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "9157c2c7030641f08f6e297e1fbbb237"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "c6cf234c44ea6fde7fdca093ca47c73d"
server_key_iv: "7270d335e543a7d515d661b7ab075964"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.062263 24848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "9157c2c7030641f08f6e297e1fbbb237"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "c6cf234c44ea6fde7fdca093ca47c73d"
server_key_iv: "7270d335e543a7d515d661b7ab075964"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.066097 24848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:33.068581 24863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.069921 24848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:33.070056 24848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "9157c2c7030641f08f6e297e1fbbb237"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "c6cf234c44ea6fde7fdca093ca47c73d"
server_key_iv: "7270d335e543a7d515d661b7ab075964"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.070178 24848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:33.085561 24848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:33.088585 24848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:33.088769 24848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:33.096686 24848 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:35921
I20260618 22:27:33.096694 24925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:35921 every 8 connection(s)
I20260618 22:27:33.097640 24848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:33.100492 24926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:33.106235 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Bootstrap starting.
I20260618 22:27:33.107445 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24848
I20260618 22:27:33.107565 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:33.107856 17358 external_mini_cluster.cc:1428] Setting key ece509666ec045f455f68ab9e06ded17
I20260618 22:27:33.109136 24926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:33.109937 24926 log.cc:824] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:33.111937 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: No bootstrap required, opened a new log
I20260618 22:27:33.115509 24926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } }
I20260618 22:27:33.115759 24926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:33.115821 24926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9157c2c7030641f08f6e297e1fbbb237, State: Initialized, Role: FOLLOWER
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:33.116245 24926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } }
I20260618 22:27:33.116379 24926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:33.116439 24926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:33.116523 24926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:33.117416 24926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } }
I20260618 22:27:33.117748 24926 leader_election.cc:302] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9157c2c7030641f08f6e297e1fbbb237; no voters:
I20260618 22:27:33.118103 24926 leader_election.cc:288] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:33.118197 24931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:33.118399 24931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [term 1 LEADER]: Becoming Leader. State: Replica: 9157c2c7030641f08f6e297e1fbbb237, State: Running, Role: LEADER
I20260618 22:27:33.118758 24931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } }
I20260618 22:27:33.119477 24926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:33.120401 24932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9157c2c7030641f08f6e297e1fbbb237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } } }
I20260618 22:27:33.120527 24932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:33.120858 24933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9157c2c7030641f08f6e297e1fbbb237. Latest consensus state: current_term: 1 leader_uuid: "9157c2c7030641f08f6e297e1fbbb237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9157c2c7030641f08f6e297e1fbbb237" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35921 } } }
I20260618 22:27:33.120959 24933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:33.123253 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.109031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.16.243.190:35921)
0618 22:27:33.109476 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.109490 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.109526 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:27:33.110217 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.110237 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.110701 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.111051 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.111843 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.112853 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.113689 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.114272 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.114430 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.116921 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.116926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.116939 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.116968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.118932 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.119423 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.119429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.119435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.119587 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.119893 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.119897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.119898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.120463 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.121757 (+ 1294us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.121924 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.122232 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
W20260618 22:27:33.124666 24946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:27:33.124730 24946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:27:33.124805 24940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:33.125483 24940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:33.130796 24940 catalog_manager.cc:1347] Generated new cluster ID: 5bf1c03e11cf40d4a52eff0b08ab25d9
I20260618 22:27:33.130879 24940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:33.151914 24940 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:33.152872 24940 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:33.158125 24940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Generated new TSK 0
I20260618 22:27:33.158885 24940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
I20260618 22:27:33.189879 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35921
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:33.302186 24952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:33.302474 24952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:33.302533 24952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:33.306639 24952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:33.306707 24952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:33.306763 24952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:33.306820 24952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:33.312131 24952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:33.313216 24952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:33.314247 24952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:33.323753 24958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:33.323753 24960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.323800 24952 server_base.cc:1034] running on GCE node
W20260618 22:27:33.323753 24957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.324460 24952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:33.325145 24952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:33.326304 24952 hybrid_clock.cc:648] HybridClock initialized: now 1781821653326283 us; error 35 us; skew 500 ppm
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:33.329144 24952 init.cc:377] Logged in from keytab as oryx/127.16.243.129@KRBTEST.COM (short username oryx)
I20260618 22:27:33.330047 24952 webserver.cc:460] Webserver started at http://127.16.243.129:36779/ using document root <none> and password file <none>
I20260618 22:27:33.330653 24952 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:33.330719 24952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:33.330930 24952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:33.333000 24952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "29bfc7a34983fa69b229bd021fd7d11a"
server_key_iv: "239c569c54a39c84a4e1b4d406a0a640"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.333559 24952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "29bfc7a34983fa69b229bd021fd7d11a"
server_key_iv: "239c569c54a39c84a4e1b4d406a0a640"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.337318 24952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:33.339798 24967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.341216 24952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:33.341328 24952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "29bfc7a34983fa69b229bd021fd7d11a"
server_key_iv: "239c569c54a39c84a4e1b4d406a0a640"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.341425 24952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:33.359786 24952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:33.363432 24952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:33.363600 24952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:33.365060 24952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:33.365120 24952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.365175 24952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:33.365213 24952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:33.375293 24952 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:44961
I20260618 22:27:33.375370 25082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:44961 every 8 connection(s)
I20260618 22:27:33.376323 24952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:33.376799 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 24952
I20260618 22:27:33.376926 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:33.377242 17358 external_mini_cluster.cc:1428] Setting key 0395ed8963a9d0439803972835fdfb30
I20260618 22:27:33.379407 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.365984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:49993 (local address 127.16.243.190:35921)
0618 22:27:33.366104 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.366107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.367284 (+ 1177us) server_negotiation.cc:407] Connection header received
0618 22:27:33.368431 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.368435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.368500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.368556 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.370327 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.370968 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.372182 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.372501 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.372594 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.375626 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.375629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.375631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.375660 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.377170 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.377757 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.377761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.377763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.377861 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.378251 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.378254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.378255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.378455 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.378545 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.379112 (+ 567us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.379252 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:33.380275 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.366598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.129:49993)
0618 22:27:33.367122 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.367168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.368163 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.368979 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.368986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.369467 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.370173 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.370192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.371094 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.371098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.372058 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.372069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.372663 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.372667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.372838 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.373480 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.373502 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.375444 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.377333 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.377338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.377353 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.377643 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.377957 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.377960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.377962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.378135 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.378582 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.378589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.378877 (+ 288us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.379037 (+ 160us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.379305 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":141,"threads_started":1}
I20260618 22:27:33.381577 25083 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35921
I20260618 22:27:33.381731 25083 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:33.382287 25083 heartbeater.cc:511] Master 127.16.243.190:35921 requested a full tablet report, sending...
I20260618 22:27:33.384097 24880 ts_manager.cc:194] Registered new tserver with Master: 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961)
I20260618 22:27:33.385648 24880 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:49993
WARNING: no policy specified for oryx/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:33.390921 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.383603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.129:57041)
0618 22:27:33.383925 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.383936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.384023 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.384315 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.384318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.384484 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.384583 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.384588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.385589 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.385592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.386220 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.386229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.386916 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.386919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.387014 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.387460 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.387473 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.387886 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.389497 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.389501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.389503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.389711 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.389957 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.389960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.389962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.390044 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.390430 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.390433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.390527 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.390651 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.390771 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260618 22:27:33.391002 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.383672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:57041 (local address 127.16.243.190:35921)
0618 22:27:33.383784 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.383789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.384000 (+ 211us) server_negotiation.cc:407] Connection header received
0618 22:27:33.384094 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.384097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.384138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.384206 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.384693 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.385473 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.386379 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.386737 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.386852 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.388008 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.388011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.388013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.388034 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.389399 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.389797 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.389801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.389803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.389880 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.390118 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.390121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.390123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.390309 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.390409 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.390717 (+ 308us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.390841 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":241,"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
I20260618 22:27:33.403347 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35921
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:33.513222 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:33.513516 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:33.513561 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:33.517714 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:33.517800 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:33.517841 25092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:33.517874 25092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:33.523654 25092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:33.524784 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:33.525748 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:33.535115 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:33.535087 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:33.535140 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.535099 25092 server_base.cc:1034] running on GCE node
I20260618 22:27:33.535687 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:33.536334 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:33.537487 25092 hybrid_clock.cc:648] HybridClock initialized: now 1781821653537476 us; error 24 us; skew 500 ppm
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:33.540124 25092 init.cc:377] Logged in from keytab as oryx/127.16.243.130@KRBTEST.COM (short username oryx)
I20260618 22:27:33.540992 25092 webserver.cc:460] Webserver started at http://127.16.243.130:39009/ using document root <none> and password file <none>
I20260618 22:27:33.541528 25092 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:33.541592 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:33.541785 25092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:33.543869 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "26f58d3c6d334f2da5446be424f89d33"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "7a6db4d978db39a6b83de2463d6e578c"
server_key_iv: "dd6a5fdcd5cacda092762c9deee0e113"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.544450 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "26f58d3c6d334f2da5446be424f89d33"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "7a6db4d978db39a6b83de2463d6e578c"
server_key_iv: "dd6a5fdcd5cacda092762c9deee0e113"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.548255 25092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:27:33.550756 25107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.551985 25092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:33.552093 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "26f58d3c6d334f2da5446be424f89d33"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "7a6db4d978db39a6b83de2463d6e578c"
server_key_iv: "dd6a5fdcd5cacda092762c9deee0e113"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.552193 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:33.562222 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:33.565289 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:33.565469 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:33.566922 25092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:33.566972 25092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.567023 25092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:33.567052 25092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.576330 25092 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:40469
I20260618 22:27:33.576428 25222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:40469 every 8 connection(s)
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:33.577425 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:33.579113 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25092
I20260618 22:27:33.579231 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:33.579509 17358 external_mini_cluster.cc:1428] Setting key 50479ef352f1138c9217c86c17447da6
I20260618 22:27:33.581395 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.567754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57963 (local address 127.16.243.190:35921)
0618 22:27:33.567900 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.567905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.568867 (+ 962us) server_negotiation.cc:407] Connection header received
0618 22:27:33.569939 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.569943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.570005 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.570103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.571987 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.572797 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.574357 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.574741 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.574862 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.577882 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.577886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.577889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.577919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.579305 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.579841 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.579846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.579848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.579943 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.580281 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.580284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.580286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.580477 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.580565 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.581107 (+ 542us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.581244 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:27:33.582520 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.568181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.130:57963)
0618 22:27:33.568620 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.568672 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.569683 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.570331 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.570341 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.570978 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.571780 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.571804 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.572938 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.572942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.574215 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.574226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.574905 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.574909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.575149 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.575814 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.575832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.577695 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.579410 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.579416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.579432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.579742 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.580037 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.580040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.580041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.580181 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.580574 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.580581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.580851 (+ 270us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.581065 (+ 214us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.581431 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260618 22:27:33.583729 25223 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35921
I20260618 22:27:33.583879 25223 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:33.584368 25223 heartbeater.cc:511] Master 127.16.243.190:35921 requested a full tablet report, sending...
I20260618 22:27:33.585392 24877 ts_manager.cc:194] Registered new tserver with Master: 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469)
I20260618 22:27:33.586128 24877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:57963
WARNING: no policy specified for oryx/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:33.593170 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.585805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.130:59569)
0618 22:27:33.586059 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.586072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.586158 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.586354 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.586356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.586621 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.586724 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.586730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.587755 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.587758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.588653 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.588664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.589118 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.589122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.589215 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.589706 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.589719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.590146 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.591851 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.591854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.591856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.592047 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.592251 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.592253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.592254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.592325 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.592692 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.592694 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.592775 (+ 81us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.592860 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.592982 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260618 22:27:33.593196 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.586055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59569 (local address 127.16.243.190:35921)
0618 22:27:33.586170 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.586174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.586186 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:27:33.586223 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.586226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.586263 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.586329 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.586825 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.587603 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.588781 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.589002 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.589077 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.590316 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.590319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.590321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.590345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.591743 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.592116 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.592118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.592119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.592186 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.592394 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.592396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.592397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.592619 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.592712 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.592922 (+ 210us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.593035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
I20260618 22:27:33.607009 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35921
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:33.717149 25232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:33.717435 25232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:33.717499 25232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:33.721768 25232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:33.721839 25232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:33.721896 25232 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:27:33.721954 25232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:33.727399 25232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.243.190:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:33.728499 25232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:33.729537 25232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:33.738832 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:33.738862 25238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.739120 25232 server_base.cc:1034] running on GCE node
W20260618 22:27:33.738866 25240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:33.739513 25232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:33.740134 25232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:33.741312 25232 hybrid_clock.cc:648] HybridClock initialized: now 1781821653741292 us; error 28 us; skew 500 ppm
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:33.744120 25232 init.cc:377] Logged in from keytab as oryx/127.16.243.131@KRBTEST.COM (short username oryx)
I20260618 22:27:33.745035 25232 webserver.cc:460] Webserver started at http://127.16.243.131:42773/ using document root <none> and password file <none>
I20260618 22:27:33.745550 25232 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:33.745635 25232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:33.745863 25232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:33.747931 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "35b3c11256044769999eb2b190aebb69"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "d840fb8dbdbca6d7663874df3ff258f5"
server_key_iv: "532677cf21a96593d1795ff75365161b"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.748425 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "35b3c11256044769999eb2b190aebb69"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "d840fb8dbdbca6d7663874df3ff258f5"
server_key_iv: "532677cf21a96593d1795ff75365161b"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.752090 25232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:27:33.754489 25247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.755790 25232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:33.755887 25232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "35b3c11256044769999eb2b190aebb69"
format_stamp: "Formatted at 2026-06-18 22:27:33 on dist-test-slave-jm5g"
server_key: "d840fb8dbdbca6d7663874df3ff258f5"
server_key_iv: "532677cf21a96593d1795ff75365161b"
server_key_version: "encryptionkey@0"
I20260618 22:27:33.755991 25232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:33.774473 25232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:33.777750 25232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:33.777956 25232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:33.779479 25232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:33.779532 25232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:33.779585 25232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:33.779614 25232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:33.790091 25232 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35885
I20260618 22:27:33.790186 25362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35885 every 8 connection(s)
I20260618 22:27:33.791043 25232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:33.792539 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.780260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:42515 (local address 127.16.243.190:35921)
0618 22:27:33.780415 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.780420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.781278 (+ 858us) server_negotiation.cc:407] Connection header received
0618 22:27:33.782303 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.782307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.782368 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.782469 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.783791 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.784411 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.785661 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.785907 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.785999 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.788960 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.788964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.788967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.788998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.790604 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.791069 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.791072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.791074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.791174 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.791500 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.791503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.791505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.791734 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.791827 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.792314 (+ 487us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.792419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:27:33.792915 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25232
I20260618 22:27:33.793005 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:33.793262 17358 external_mini_cluster.cc:1428] Setting key f26ad1a797968cfd4c125ef515d872df
I20260618 22:27:33.793454 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.780640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.131:42515)
0618 22:27:33.781105 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.781156 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.782060 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.782691 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.782700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.783095 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.783646 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.783661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.784548 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.784550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.785547 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.785557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.786071 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.786075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.786232 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.786869 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.786905 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.788754 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.790703 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.790709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.790725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.790974 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.791260 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.791263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.791265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.791398 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.791826 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.791833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.792103 (+ 270us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.792271 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.792529 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260618 22:27:33.794703 25363 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35921
I20260618 22:27:33.794852 25363 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:33.795394 25363 heartbeater.cc:511] Master 127.16.243.190:35921 requested a full tablet report, sending...
I20260618 22:27:33.796290 24877 ts_manager.cc:194] Registered new tserver with Master: 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:33.796821 24877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:42515
I20260618 22:27:33.803113 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.797298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:52735 (local address 127.16.243.190:35921)
0618 22:27:33.797410 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.797414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.797582 (+ 168us) server_negotiation.cc:407] Connection header received
0618 22:27:33.797694 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.797696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.797740 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.797795 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.798229 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.798759 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.799508 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.799728 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.799802 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.800786 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.800789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.800791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.800805 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.801860 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.802192 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.802195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.802196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.802261 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.802444 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.802445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.802447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.802638 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.802708 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.802905 (+ 197us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.802992 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:27:33.803128 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.797229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.16.243.131:52735)
0618 22:27:33.797505 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.797519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.797614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.797830 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.797833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.798030 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:33.798138 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.798145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.798866 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.798869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.799408 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.799415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.799813 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.799816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.799877 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.800366 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:33.800378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:33.800694 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:33.801942 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.801945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.801946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.802124 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.802319 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:33.802321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:33.802322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:33.802387 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:33.802698 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:33.802700 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:33.802780 (+ 80us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.802860 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.802966 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260618 22:27:33.806806 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:27:33.829433 24929 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:33.823294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.16.243.190:35921)
0618 22:27:33.823424 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.823428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.823441 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:33.823491 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.823493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.823533 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.823605 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.824379 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.824935 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.825813 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.826027 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.826114 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.829282 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38478: BlockingRecv error: recv got EOF from 127.0.0.1:38478 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 22:27:33 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:33.838284 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.830900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.16.243.190:35921)
0618 22:27:33.830996 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.830999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.831120 (+ 121us) server_negotiation.cc:407] Connection header received
0618 22:27:33.831231 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.831233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.831270 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.831336 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:33.831869 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.832373 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.833091 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.833303 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.833367 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.835507 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:33.835511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:33.835513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:33.835537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:33.836928 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.837348 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.837351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.837352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.837422 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:33.837617 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:33.837619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:33.837620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:33.837785 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:33.837878 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.838047 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.838136 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:27:33.841262 24877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:33.844317 24877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:33.858829 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.853479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.16.243.130:40469)
0618 22:27:33.853921 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.853933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.853954 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:27:33.854024 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.854029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.854222 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.854365 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:33.854728 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.855895 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.857384 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.858124 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.858265 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.858400 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.858589 (+ 189us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.858695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":95,"threads_started":1}
I20260618 22:27:33.858806 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.853332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.0.0.1:56996)
0618 22:27:33.853612 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.853624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.853703 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.854370 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.854375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.854411 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:33.854609 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.854618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.856000 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.856003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.857287 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.857294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.858255 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.858258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.858347 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.858383 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.858491 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.858646 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":102,"spinlock_wait_cycles":7296,"thread_start_us":81,"threads_started":1}
I20260618 22:27:33.859046 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.852643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35885 (local address 127.0.0.1:38878)
0618 22:27:33.852985 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.853030 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.853147 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.854511 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.854514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.854527 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:33.854673 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.854680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.855777 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.855780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.857213 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.857223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.858654 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.858658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.858759 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.858781 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.858847 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.858913 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":73,"thread_start_us":92,"threads_started":1}
I20260618 22:27:33.859917 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.852859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44961 (local address 127.0.0.1:56984)
0618 22:27:33.853587 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:33.853600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:33.853675 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:33.854381 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:33.854384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:33.854414 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:33.854881 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.854890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.856123 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.856126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.857284 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:33.857294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.859537 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.859541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:33.859629 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.859661 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:27:33.859724 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:33.859788 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":351,"spinlock_wait_cycles":3584,"thread_start_us":70,"threads_started":1}
I20260618 22:27:33.860044 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.853336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.16.243.129:44961)
0618 22:27:33.854023 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.854035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.854051 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:27:33.854107 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.854111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.854247 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.854499 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:33.855000 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.856032 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.858458 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.859433 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.859596 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.859765 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.859826 (+ 61us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.859916 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":81,"threads_started":1}
I20260618 22:27:33.861991 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 68e232289736407bb3398de7166a45b9 (DEFAULT_TABLE table=test-table [id=01e018ae9044424e9a955b9955c5f0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:33.862317 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 68e232289736407bb3398de7166a45b9 (DEFAULT_TABLE table=test-table [id=01e018ae9044424e9a955b9955c5f0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:33.862854 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:33.852792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.16.243.131:35885)
0618 22:27:33.853902 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:33.853915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:33.853937 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:27:33.854009 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:33.854015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:33.854232 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:33.854376 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:33.854773 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.855676 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.857322 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:33.858124 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:33.858275 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:33.858400 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:33.862596 (+ 4196us) server_negotiation.cc:299] Negotiation successful
0618 22:27:33.862707 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":129,"threads_started":1}
I20260618 22:27:33.862999 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e232289736407bb3398de7166a45b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:33.863346 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e232289736407bb3398de7166a45b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:33.865538 25297 tablet_service.cc:1467] Processing CreateTablet for tablet 68e232289736407bb3398de7166a45b9 (DEFAULT_TABLE table=test-table [id=01e018ae9044424e9a955b9955c5f0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:33.866611 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e232289736407bb3398de7166a45b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:33.869858 25391 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Bootstrap starting.
I20260618 22:27:33.871515 25392 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Bootstrap starting.
I20260618 22:27:33.872359 25391 tablet_bootstrap.cc:654] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:33.873090 25391 log.cc:824] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:33.873687 25392 tablet_bootstrap.cc:654] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:33.874691 25392 log.cc:824] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:33.875144 25391 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: No bootstrap required, opened a new log
I20260618 22:27:33.875339 25391 ts_tablet_manager.cc:1397] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:27:33.875660 25394 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Bootstrap starting.
I20260618 22:27:33.876695 25392 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: No bootstrap required, opened a new log
I20260618 22:27:33.876881 25392 ts_tablet_manager.cc:1397] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260618 22:27:33.877832 25394 tablet_bootstrap.cc:654] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:33.878670 25394 log.cc:824] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:33.878520 25391 raft_consensus.cc:348] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.878803 25391 raft_consensus.cc:374] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:33.878885 25391 raft_consensus.cc:729] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Initialized, Role: FOLLOWER
I20260618 22:27:33.879405 25391 consensus_queue.cc:260] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.880125 25223 heartbeater.cc:503] Master 127.16.243.190:35921 was elected leader, sending a full tablet report...
I20260618 22:27:33.880434 25391 ts_tablet_manager.cc:1428] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:33.880577 25392 raft_consensus.cc:348] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.880869 25392 raft_consensus.cc:374] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:33.880934 25392 raft_consensus.cc:729] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Initialized, Role: FOLLOWER
I20260618 22:27:33.881402 25392 consensus_queue.cc:260] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.881980 25394 tablet_bootstrap.cc:492] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: No bootstrap required, opened a new log
I20260618 22:27:33.882030 25083 heartbeater.cc:503] Master 127.16.243.190:35921 was elected leader, sending a full tablet report...
I20260618 22:27:33.882236 25394 ts_tablet_manager.cc:1397] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:27:33.884438 25392 ts_tablet_manager.cc:1428] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260618 22:27:33.885973 25394 raft_consensus.cc:348] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.886200 25394 raft_consensus.cc:374] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:33.886255 25394 raft_consensus.cc:729] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Initialized, Role: FOLLOWER
I20260618 22:27:33.886716 25394 consensus_queue.cc:260] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:33.887368 25363 heartbeater.cc:503] Master 127.16.243.190:35921 was elected leader, sending a full tablet report...
I20260618 22:27:33.887578 25394 ts_tablet_manager.cc:1428] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:27:34.042165 25364 tablet.cc:2369] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:34.078763 25224 tablet.cc:2369] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:27:34.127909 25084 tablet.cc:2369] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:27:34.199234 25398 raft_consensus.cc:482] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:34.199465 25398 raft_consensus.cc:504] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:34.200593 25398 leader_election.cc:288] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885), 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469)
I20260618 22:27:34.204404 25397 raft_consensus.cc:482] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:34.204623 25397 raft_consensus.cc:504] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:34.205534 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.201069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:41925 (local address 127.16.243.131:35885)
0618 22:27:34.201213 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.201220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.201308 (+ 88us) server_negotiation.cc:407] Connection header received
0618 22:27:34.201470 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.201474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.201526 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.201621 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.201895 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.202731 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.204489 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.204983 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.205098 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.205153 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.205329 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.205415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:27:34.205629 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.201408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:33283 (local address 127.16.243.130:40469)
0618 22:27:34.201564 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.201569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.201684 (+ 115us) server_negotiation.cc:407] Connection header received
0618 22:27:34.201777 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.201779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.201826 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.201902 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.202109 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.202841 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.204440 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.204998 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.205119 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.205166 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.205452 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.205529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:27:34.205552 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.200875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35885 (local address 127.16.243.129:41925)
0618 22:27:34.201203 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.201221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.201363 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.201653 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.201656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.201682 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.201793 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.201800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.202861 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.202864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.204314 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.204326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.205098 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.205102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.205196 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.205221 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.205294 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.205378 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":77,"thread_start_us":109,"threads_started":1}
I20260618 22:27:34.205994 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.201306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.16.243.129:33283)
0618 22:27:34.201591 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.201604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.201694 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.201919 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.201921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.201936 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.202012 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.202016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.202949 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.202952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.204314 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.204326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.205220 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.205223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.205326 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.205345 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.205800 (+ 455us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.205875 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260618 22:27:34.205977 25397 leader_election.cc:288] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885), 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961)
I20260618 22:27:34.206497 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69" is_pre_election: true
I20260618 22:27:34.206832 25317 raft_consensus.cc:2454] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddf01d4c1c942aeaaaaea8cec9c63cd in term 0.
I20260618 22:27:34.206861 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f58d3c6d334f2da5446be424f89d33" is_pre_election: true
I20260618 22:27:34.207026 25177 raft_consensus.cc:2454] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ddf01d4c1c942aeaaaaea8cec9c63cd in term 0.
I20260618 22:27:34.207387 24971 leader_election.cc:302] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b3c11256044769999eb2b190aebb69, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.207891 25398 raft_consensus.cc:2781] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:34.207989 25398 raft_consensus.cc:482] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:34.208042 25398 raft_consensus.cc:3037] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.209336 25398 raft_consensus.cc:504] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:34.209695 25398 leader_election.cc:288] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [CANDIDATE]: Term 1 election: Requested vote from peers 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885), 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469)
I20260618 22:27:34.210006 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69"
I20260618 22:27:34.210127 25317 raft_consensus.cc:3037] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.210464 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f58d3c6d334f2da5446be424f89d33"
I20260618 22:27:34.210523 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.206708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:52055 (local address 127.16.243.129:44961)
0618 22:27:34.206813 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.206816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.206825 (+ 9us) server_negotiation.cc:407] Connection header received
0618 22:27:34.206855 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.206857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.206890 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.206947 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.207209 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.207869 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.209378 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.209894 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.209974 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.210022 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.210345 (+ 323us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.210414 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:27:34.210721 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.207517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:54825 (local address 127.16.243.131:35885)
0618 22:27:34.207624 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.207627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.207710 (+ 83us) server_negotiation.cc:407] Connection header received
0618 22:27:34.207814 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.207817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.207860 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.207931 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.208229 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.208688 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.209670 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.210163 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.210262 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.210308 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.210505 (+ 197us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.210598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:34.210748 25177 raft_consensus.cc:3037] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.210907 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.206315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44961 (local address 127.16.243.130:52055)
0618 22:27:34.206516 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.206529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.206654 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.206971 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.206974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.206991 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.207098 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.207104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.207994 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.207997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.209244 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.209255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.210110 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.210114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.210209 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.210232 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.210665 (+ 433us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.210753 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":37,"thread_start_us":70,"threads_started":1}
I20260618 22:27:34.211045 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.207203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35885 (local address 127.16.243.130:54825)
0618 22:27:34.207648 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.207661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.207738 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.207946 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.207949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.207964 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.208129 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.208135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.208790 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.208793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.209554 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.209564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.210268 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.210271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.210375 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.210397 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.210830 (+ 433us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.210921 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":28,"thread_start_us":69,"threads_started":1}
I20260618 22:27:34.211804 25177 raft_consensus.cc:2454] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddf01d4c1c942aeaaaaea8cec9c63cd in term 1.
I20260618 22:27:34.211861 25317 raft_consensus.cc:2454] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ddf01d4c1c942aeaaaaea8cec9c63cd in term 1.
I20260618 22:27:34.212158 24971 leader_election.cc:302] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.212451 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" is_pre_election: true
I20260618 22:27:34.212603 25398 raft_consensus.cc:2781] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:34.212683 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e232289736407bb3398de7166a45b9" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69" is_pre_election: true
I20260618 22:27:34.212854 25317 raft_consensus.cc:2379] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26f58d3c6d334f2da5446be424f89d33 in current term 1: Already voted for candidate 8ddf01d4c1c942aeaaaaea8cec9c63cd in this term.
I20260618 22:27:34.212844 25398 raft_consensus.cc:686] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 LEADER]: Becoming Leader. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Running, Role: LEADER
I20260618 22:27:34.213204 25398 consensus_queue.cc:237] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } }
I20260618 22:27:34.216012 25108 leader_election.cc:302] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33; no voters: 35b3c11256044769999eb2b190aebb69, 8ddf01d4c1c942aeaaaaea8cec9c63cd
I20260618 22:27:34.216490 25397 raft_consensus.cc:2726] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260618 22:27:34.216694 24880 catalog_manager.cc:5495] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129). New cstate: current_term: 1 leader_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } health_report { overall_health: HEALTHY } } }
I20260618 22:27:34.309599 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.306534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.16.243.129:44961)
0618 22:27:34.306690 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.306694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.306707 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:34.306778 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.306780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.306835 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.306917 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.307107 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.307602 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.308383 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.308597 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.308672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.308873 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.309308 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.309389 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.309418 (+ 29us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.309465 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260618 22:27:34.315394 25317 raft_consensus.cc:1261] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Refusing update from remote peer 8ddf01d4c1c942aeaaaaea8cec9c63cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.315433 25177 raft_consensus.cc:1261] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Refusing update from remote peer 8ddf01d4c1c942aeaaaaea8cec9c63cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.316025 25404 consensus_queue.cc:1035] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.316202 25398 consensus_queue.cc:1035] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.322388 25407 mvcc.cc:205] Tried to move back new op lower bound from 7298341496068337664 to 7298341495660359680. Current Snapshot: MvccSnapshot[applied={T|T < 7298341496068337664}]
I20260618 22:27:34.323680 25408 mvcc.cc:205] Tried to move back new op lower bound from 7298341496068337664 to 7298341495660359680. Current Snapshot: MvccSnapshot[applied={T|T < 7298341496068337664}]
I20260618 22:27:34.325659 25409 mvcc.cc:205] Tried to move back new op lower bound from 7298341496068337664 to 7298341495660359680. Current Snapshot: MvccSnapshot[applied={T|T < 7298341496068337664}]
I20260618 22:27:34.331363 24880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.331527 24880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.334013 24880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Sending DeleteTablet for 3 replicas of tablet 68e232289736407bb3398de7166a45b9
I20260618 22:27:34.334827 25017 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e232289736407bb3398de7166a45b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56984
I20260618 22:27:34.334868 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e232289736407bb3398de7166a45b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260618 22:27:34.335310 25416 tablet_replica.cc:330] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: stopping tablet replica
I20260618 22:27:34.335326 25157 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e232289736407bb3398de7166a45b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56996
I20260618 22:27:34.335664 25416 raft_consensus.cc:2229] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:34.336009 25416 raft_consensus.cc:2258] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.336349 25418 tablet_replica.cc:330] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: stopping tablet replica
I20260618 22:27:34.336385 25417 tablet_replica.cc:330] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: stopping tablet replica
I20260618 22:27:34.336536 24880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:27:34.336686 25417 raft_consensus.cc:2229] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:34.336892 25418 raft_consensus.cc:2229] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260618 22:27:34.336962 24880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:34.337083 25417 raft_consensus.cc:2258] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.337352 25418 raft_consensus.cc:2258] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.337818 25416 ts_tablet_manager.cc:1905] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.338397 25417 ts_tablet_manager.cc:1905] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.338781 25418 ts_tablet_manager.cc:1905] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.340481 25416 ts_tablet_manager.cc:1918] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.340587 25416 log.cc:1196] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/68e232289736407bb3398de7166a45b9
I20260618 22:27:34.340895 25416 ts_tablet_manager.cc:1939] T 68e232289736407bb3398de7166a45b9 P 35b3c11256044769999eb2b190aebb69: Deleting consensus metadata
I20260618 22:27:34.341224 25417 ts_tablet_manager.cc:1918] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.341326 25417 log.cc:1196] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/68e232289736407bb3398de7166a45b9
I20260618 22:27:34.341594 25417 ts_tablet_manager.cc:1939] T 68e232289736407bb3398de7166a45b9 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting consensus metadata
I20260618 22:27:34.341846 25418 ts_tablet_manager.cc:1918] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.341974 25418 log.cc:1196] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/68e232289736407bb3398de7166a45b9
I20260618 22:27:34.342300 25418 ts_tablet_manager.cc:1939] T 68e232289736407bb3398de7166a45b9 P 26f58d3c6d334f2da5446be424f89d33: Deleting consensus metadata
I20260618 22:27:34.342624 25297 tablet_service.cc:1467] Processing CreateTablet for tablet 53f8cc845f6d4b669617218d86381101 (DEFAULT_TABLE table=test-table [id=979b2e8521c647598b661027d4286544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.342880 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f8cc845f6d4b669617218d86381101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.343469 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 53f8cc845f6d4b669617218d86381101 (DEFAULT_TABLE table=test-table [id=979b2e8521c647598b661027d4286544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.343710 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f8cc845f6d4b669617218d86381101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.344738 25394 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Bootstrap starting.
I20260618 22:27:34.344915 24867 catalog_manager.cc:4841] TS 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469): tablet 68e232289736407bb3398de7166a45b9 (table test-table [id=01e018ae9044424e9a955b9955c5f0fa]) successfully deleted
I20260618 22:27:34.345079 24867 catalog_manager.cc:4841] TS 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885): tablet 68e232289736407bb3398de7166a45b9 (table test-table [id=01e018ae9044424e9a955b9955c5f0fa]) successfully deleted
I20260618 22:27:34.345580 25394 tablet_bootstrap.cc:654] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.345839 25391 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Bootstrap starting.
I20260618 22:27:34.346129 24864 catalog_manager.cc:4841] TS 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961): tablet 68e232289736407bb3398de7166a45b9 (table test-table [id=01e018ae9044424e9a955b9955c5f0fa]) successfully deleted
I20260618 22:27:34.346714 25391 tablet_bootstrap.cc:654] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.346973 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 53f8cc845f6d4b669617218d86381101 (DEFAULT_TABLE table=test-table [id=979b2e8521c647598b661027d4286544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.347226 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f8cc845f6d4b669617218d86381101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.348011 25391 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: No bootstrap required, opened a new log
I20260618 22:27:34.348086 25391 ts_tablet_manager.cc:1397] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:34.348464 25391 raft_consensus.cc:348] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.348567 25391 raft_consensus.cc:374] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.348600 25391 raft_consensus.cc:729] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.348717 25391 consensus_queue.cc:260] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.348950 25391 ts_tablet_manager.cc:1428] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.349517 25392 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Bootstrap starting.
I20260618 22:27:34.350414 25394 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: No bootstrap required, opened a new log
I20260618 22:27:34.350449 25392 tablet_bootstrap.cc:654] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.350504 25394 ts_tablet_manager.cc:1397] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:27:34.351013 25394 raft_consensus.cc:348] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.351147 25394 raft_consensus.cc:374] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.351181 25394 raft_consensus.cc:729] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.351320 25394 consensus_queue.cc:260] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.351573 25394 ts_tablet_manager.cc:1428] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.351588 25392 tablet_bootstrap.cc:492] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: No bootstrap required, opened a new log
I20260618 22:27:34.351655 25392 ts_tablet_manager.cc:1397] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 22:27:34.351989 25392 raft_consensus.cc:348] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.352079 25392 raft_consensus.cc:374] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.352110 25392 raft_consensus.cc:729] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.352264 25392 consensus_queue.cc:260] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.352504 25392 ts_tablet_manager.cc:1428] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:27:34.397990 25399 raft_consensus.cc:482] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:34.398137 25399 raft_consensus.cc:504] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.399255 25399 leader_election.cc:288] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469)
I20260618 22:27:34.403266 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.399580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:52873 (local address 127.16.243.129:44961)
0618 22:27:34.399699 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.399704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.399876 (+ 172us) server_negotiation.cc:407] Connection header received
0618 22:27:34.400101 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.400104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.400152 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.400210 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.400641 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.401281 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.402328 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.402745 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.402852 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.402906 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.403088 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.403157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:27:34.403290 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.399480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44961 (local address 127.16.243.131:52873)
0618 22:27:34.399785 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.399799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.399973 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.400328 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.400331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.400350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.400513 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.400521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.401393 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.401395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.402190 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.402198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.402879 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.402883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.402977 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.402994 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.403058 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.403129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"spinlock_wait_cycles":16512,"thread_start_us":69,"threads_started":1}
I20260618 22:27:34.403949 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.399511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.16.243.131:37357)
0618 22:27:34.399854 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.399866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.399973 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.400352 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.400356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.400375 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:34.400532 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.400539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.401559 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.401562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.402772 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.402783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.403574 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.403578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.403670 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.403692 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.403756 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.403815 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":29,"spinlock_wait_cycles":20096,"thread_start_us":101,"threads_started":1}
I20260618 22:27:34.404068 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f8cc845f6d4b669617218d86381101" candidate_uuid: "35b3c11256044769999eb2b190aebb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" is_pre_election: true
I20260618 22:27:34.404167 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.399899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:37357 (local address 127.16.243.130:40469)
0618 22:27:34.400016 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.400020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.400034 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:34.400104 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.400107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.400158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.400232 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:34.400656 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.401423 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.402912 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.403475 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.403588 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.403634 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.403958 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.404042 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:27:34.404242 25037 raft_consensus.cc:2454] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35b3c11256044769999eb2b190aebb69 in term 0.
I20260618 22:27:34.404618 25248 leader_election.cc:302] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b3c11256044769999eb2b190aebb69, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.404655 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f8cc845f6d4b669617218d86381101" candidate_uuid: "35b3c11256044769999eb2b190aebb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f58d3c6d334f2da5446be424f89d33" is_pre_election: true
I20260618 22:27:34.404785 25177 raft_consensus.cc:2454] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35b3c11256044769999eb2b190aebb69 in term 0.
I20260618 22:27:34.404879 25399 raft_consensus.cc:2781] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:34.404938 25399 raft_consensus.cc:482] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:34.404983 25399 raft_consensus.cc:3037] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.405938 25399 raft_consensus.cc:504] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.406268 25399 leader_election.cc:288] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469)
I20260618 22:27:34.406541 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f8cc845f6d4b669617218d86381101" candidate_uuid: "35b3c11256044769999eb2b190aebb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
I20260618 22:27:34.406682 25037 raft_consensus.cc:3037] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.406656 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f8cc845f6d4b669617218d86381101" candidate_uuid: "35b3c11256044769999eb2b190aebb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f58d3c6d334f2da5446be424f89d33"
I20260618 22:27:34.406761 25177 raft_consensus.cc:3037] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.407596 25037 raft_consensus.cc:2454] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35b3c11256044769999eb2b190aebb69 in term 1.
I20260618 22:27:34.407837 25177 raft_consensus.cc:2454] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35b3c11256044769999eb2b190aebb69 in term 1.
I20260618 22:27:34.407860 25248 leader_election.cc:302] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b3c11256044769999eb2b190aebb69, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.408031 25399 raft_consensus.cc:2781] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:34.408254 25399 raft_consensus.cc:686] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 1 LEADER]: Becoming Leader. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Running, Role: LEADER
I20260618 22:27:34.408499 25399 consensus_queue.cc:237] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.410604 24877 catalog_manager.cc:5495] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35b3c11256044769999eb2b190aebb69 (127.16.243.131). New cstate: current_term: 1 leader_uuid: "35b3c11256044769999eb2b190aebb69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } health_report { overall_health: HEALTHY } } }
Jun 18 22:27:34 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821653, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.190@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:34.445173 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.436177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35921 (local address 127.0.0.1:38482)
0618 22:27:34.436531 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.436546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.436674 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.436922 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.436925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.437453 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:34.437571 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.437578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.438308 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.438311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.439137 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.439148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.439576 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.439580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.439664 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.440263 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:34.440285 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:34.441809 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:34.443634 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:34.443642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:34.443647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:34.443889 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:34.444172 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:34.444174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:34.444176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:34.444250 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:34.444674 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:34.444679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:34.444778 (+ 99us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.444900 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.445035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":136,"threads_started":1}
I20260618 22:27:34.445454 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.436274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.16.243.190:35921)
0618 22:27:34.436509 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.436518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.436621 (+ 103us) server_negotiation.cc:407] Connection header received
0618 22:27:34.436757 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.436760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.436800 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.436873 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:34.437679 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.438197 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.439240 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.439462 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.439533 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.441954 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:34.441959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:34.441961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:34.441989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:34.443531 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:34.443987 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:34.443990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:34.443992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:34.444087 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:34.444331 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:34.444334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:34.444336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:34.444579 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:34.444680 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.444982 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.445113 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260618 22:27:34.449087 24877 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:38482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:27:34.449766 24877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:34.454360 25017 tablet_service.cc:1467] Processing CreateTablet for tablet a4f87352a72449fdbb96376ba1531e8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6745aae190a40b78abcb8db58ab1cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:34.454439 25157 tablet_service.cc:1467] Processing CreateTablet for tablet a4f87352a72449fdbb96376ba1531e8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6745aae190a40b78abcb8db58ab1cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:34.454806 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f87352a72449fdbb96376ba1531e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.454870 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f87352a72449fdbb96376ba1531e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.457238 25392 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Bootstrap starting.
I20260618 22:27:34.457588 25297 tablet_service.cc:1467] Processing CreateTablet for tablet a4f87352a72449fdbb96376ba1531e8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6745aae190a40b78abcb8db58ab1cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:27:34.457892 25391 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: Bootstrap starting.
I20260618 22:27:34.457949 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f87352a72449fdbb96376ba1531e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.458209 25392 tablet_bootstrap.cc:654] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.458951 25391 tablet_bootstrap.cc:654] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.460362 25391 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: No bootstrap required, opened a new log
I20260618 22:27:34.460455 25391 ts_tablet_manager.cc:1397] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:34.460624 25394 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: Bootstrap starting.
I20260618 22:27:34.460932 25391 raft_consensus.cc:348] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.461043 25391 raft_consensus.cc:374] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.461076 25391 raft_consensus.cc:729] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.461196 25391 consensus_queue.cc:260] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.461524 25394 tablet_bootstrap.cc:654] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.461540 25391 ts_tablet_manager.cc:1428] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.461504 25397 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.461634 25397 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.462806 25392 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: No bootstrap required, opened a new log
I20260618 22:27:34.462891 25392 ts_tablet_manager.cc:1397] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:27:34.463284 25392 raft_consensus.cc:348] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.463385 25392 raft_consensus.cc:374] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.463426 25392 raft_consensus.cc:729] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.463529 25392 consensus_queue.cc:260] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.463797 25392 ts_tablet_manager.cc:1428] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:34.463968 25404 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.464320 25404 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.465565 25394 tablet_bootstrap.cc:492] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: No bootstrap required, opened a new log
I20260618 22:27:34.465653 25394 ts_tablet_manager.cc:1397] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 22:27:34.466087 25394 raft_consensus.cc:348] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.466202 25394 raft_consensus.cc:374] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.466234 25394 raft_consensus.cc:729] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.466384 25394 consensus_queue.cc:260] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.466670 25394 ts_tablet_manager.cc:1428] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.467361 25399 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.467465 25399 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.493642 25397 raft_consensus.cc:482] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:34.493775 25397 raft_consensus.cc:504] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.494200 25397 leader_election.cc:288] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:34.494643 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f87352a72449fdbb96376ba1531e8a" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" is_pre_election: true
I20260618 22:27:34.494798 25037 raft_consensus.cc:2454] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:34.494807 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f87352a72449fdbb96376ba1531e8a" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69" is_pre_election: true
I20260618 22:27:34.494958 25317 raft_consensus.cc:2454] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:34.495091 25108 leader_election.cc:302] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.495309 25397 raft_consensus.cc:2781] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:34.495368 25397 raft_consensus.cc:482] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:34.495415 25397 raft_consensus.cc:3037] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.496557 25397 raft_consensus.cc:504] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.496932 25397 leader_election.cc:288] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:34.497251 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f87352a72449fdbb96376ba1531e8a" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
I20260618 22:27:34.497368 25037 raft_consensus.cc:3037] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.497604 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f87352a72449fdbb96376ba1531e8a" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69"
I20260618 22:27:34.497730 25317 raft_consensus.cc:3037] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.498450 25037 raft_consensus.cc:2454] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:34.498775 25108 leader_election.cc:302] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.498835 25317 raft_consensus.cc:2454] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:34.498945 25397 raft_consensus.cc:2781] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:34.499191 25397 raft_consensus.cc:686] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [term 1 LEADER]: Becoming Leader. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Running, Role: LEADER
I20260618 22:27:34.499444 25397 consensus_queue.cc:237] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.500826 25436 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: TxnStatusTablet state changed. Reason: New leader 26f58d3c6d334f2da5446be424f89d33. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.500919 25436 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:34.501190 25438 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:27:34.501353 25438 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:27:34.501890 24880 catalog_manager.cc:5495] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:34.521179 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.517578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.0.0.1:57002)
0618 22:27:34.517725 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.517741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.517850 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.518138 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.518140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.518148 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:34.518246 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.518251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.518991 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.518995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.519590 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.519597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.520056 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.520060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.520141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.520184 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.520868 (+ 684us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.520904 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.520964 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.521049 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 22:27:34.521179 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.517694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.16.243.130:40469)
0618 22:27:34.517823 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.517828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.517842 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:34.517988 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.517991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.518044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.518129 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.518340 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.518895 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.519693 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.519960 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.520046 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.520313 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.520775 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.520882 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.520981 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.521037 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:27:34.524001 25317 raft_consensus.cc:1261] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.524161 25037 raft_consensus.cc:1261] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.524729 25436 consensus_queue.cc:1035] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.524945 25397 consensus_queue.cc:1035] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.526537 25404 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: TxnStatusTablet state changed. Reason: New leader 26f58d3c6d334f2da5446be424f89d33. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.527206 25404 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.528115 25436 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.528219 25436 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:34.529991 25404 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.530097 25404 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 8ddf01d4c1c942aeaaaaea8cec9c63cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.531845 25399 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: TxnStatusTablet state changed. Reason: New leader 26f58d3c6d334f2da5446be424f89d33. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.531965 25399 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.535583 25397 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.535708 25397 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 26f58d3c6d334f2da5446be424f89d33: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:27:34.536856 25399 tablet_replica.cc:439] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } } }
I20260618 22:27:34.536973 25399 tablet_replica.cc:442] T a4f87352a72449fdbb96376ba1531e8a P 35b3c11256044769999eb2b190aebb69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:27:34.540771 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.536709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.16.243.131:35885)
0618 22:27:34.536878 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.536882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.536894 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:27:34.536963 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.536966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.537024 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.537130 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.537542 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.538330 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.539210 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.539547 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.539649 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.539836 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.540405 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.540517 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.540547 (+ 30us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.540603 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260618 22:27:34.551412 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.547944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:52187 (local address 127.16.243.130:40469)
0618 22:27:34.548072 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.548077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.548186 (+ 109us) server_negotiation.cc:407] Connection header received
0618 22:27:34.548294 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.548297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.548347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.548424 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.548611 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.549398 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.550208 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.550607 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.550731 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.550972 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.551068 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.551139 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.551213 (+ 74us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.551270 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:27:34.551786 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.547828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.16.243.131:52187)
0618 22:27:34.548111 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.548123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.548211 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.548427 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.548429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.548435 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:34.548534 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.548538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.549483 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.549485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.550111 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.550118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.550715 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.550718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.550794 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.550851 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.551141 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.551145 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.551189 (+ 44us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.551257 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
I20260618 22:27:34.558166 25037 raft_consensus.cc:1261] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Refusing update from remote peer 35b3c11256044769999eb2b190aebb69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.558167 25177 raft_consensus.cc:1261] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Refusing update from remote peer 35b3c11256044769999eb2b190aebb69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.558631 25399 consensus_queue.cc:1035] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.558816 25424 consensus_queue.cc:1035] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.580184 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.576011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:33497 (local address 127.16.243.131:35885)
0618 22:27:34.576141 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.576145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.576182 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:27:34.576398 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.576401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.576451 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.576526 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.576781 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.577565 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.578885 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.579232 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.579302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.579706 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.579801 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.579842 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.579990 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.580048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:34.580564 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.575766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35885 (local address 127.16.243.130:33497)
0618 22:27:34.576094 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:34.576110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:34.576295 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:34.576546 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:34.576549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:34.576556 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:34.576678 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.576683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.577669 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.577672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.578751 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:34.578762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.579406 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.579409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:34.579504 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.579553 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.579898 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.579903 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:34.580336 (+ 433us) client_negotiation.cc:240] Negotiation successful
0618 22:27:34.580422 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260618 22:27:34.617314 24880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.617496 24880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.619905 24880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Sending DeleteTablet for 3 replicas of tablet 53f8cc845f6d4b669617218d86381101
I20260618 22:27:34.620661 25157 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f8cc845f6d4b669617218d86381101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56996
I20260618 22:27:34.620771 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f8cc845f6d4b669617218d86381101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260618 22:27:34.620872 25418 tablet_replica.cc:330] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: stopping tablet replica
I20260618 22:27:34.620986 25416 tablet_replica.cc:330] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: stopping tablet replica
I20260618 22:27:34.620985 25418 raft_consensus.cc:2229] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:34.621101 25418 raft_consensus.cc:2258] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.621104 25416 raft_consensus.cc:2229] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:34.621424 25416 raft_consensus.cc:2258] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.622133 25416 ts_tablet_manager.cc:1905] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.622134 25418 ts_tablet_manager.cc:1905] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.625005 25418 ts_tablet_manager.cc:1918] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:34.625005 25416 ts_tablet_manager.cc:1918] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:34.625084 25418 log.cc:1196] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/53f8cc845f6d4b669617218d86381101
I20260618 22:27:34.625084 25416 log.cc:1196] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/53f8cc845f6d4b669617218d86381101
I20260618 22:27:34.625319 25418 ts_tablet_manager.cc:1939] T 53f8cc845f6d4b669617218d86381101 P 26f58d3c6d334f2da5446be424f89d33: Deleting consensus metadata
I20260618 22:27:34.625319 25416 ts_tablet_manager.cc:1939] T 53f8cc845f6d4b669617218d86381101 P 35b3c11256044769999eb2b190aebb69: Deleting consensus metadata
I20260618 22:27:34.625574 25017 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f8cc845f6d4b669617218d86381101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56984
I20260618 22:27:34.625845 25417 tablet_replica.cc:330] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: stopping tablet replica
I20260618 22:27:34.625985 25417 raft_consensus.cc:2229] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:34.626106 25417 raft_consensus.cc:2258] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.626394 24867 catalog_manager.cc:4841] TS 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469): tablet 53f8cc845f6d4b669617218d86381101 (table test-table [id=979b2e8521c647598b661027d4286544]) successfully deleted
I20260618 22:27:34.626899 25417 ts_tablet_manager.cc:1905] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.629266 25417 ts_tablet_manager.cc:1918] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:34.629339 25417 log.cc:1196] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/53f8cc845f6d4b669617218d86381101
I20260618 22:27:34.629541 25417 ts_tablet_manager.cc:1939] T 53f8cc845f6d4b669617218d86381101 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting consensus metadata
I20260618 22:27:34.630213 24864 catalog_manager.cc:4841] TS 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961): tablet 53f8cc845f6d4b669617218d86381101 (table test-table [id=979b2e8521c647598b661027d4286544]) successfully deleted
I20260618 22:27:34.630403 24867 catalog_manager.cc:4841] TS 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885): tablet 53f8cc845f6d4b669617218d86381101 (table test-table [id=979b2e8521c647598b661027d4286544]) successfully deleted
Jun 18 22:27:34 dist-test-slave-jm5g krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:27:34 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:34 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:27:34.655774 25432 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:34.647743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.16.243.190:35921)
0618 22:27:34.647875 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.647879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.648156 (+ 277us) server_negotiation.cc:407] Connection header received
0618 22:27:34.648303 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.648307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.648362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.648452 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:34.649269 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.650107 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.651391 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.651751 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.651869 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.655623 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38498: BlockingRecv error: recv got EOF from 127.0.0.1:38498 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 18 22:27:34 dist-test-slave-jm5g krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM
I20260618 22:27:34.668282 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.658171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.16.243.190:35921)
0618 22:27:34.658328 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.658332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.658374 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:27:34.658421 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.658424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.658470 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.658544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:34.659339 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.660157 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.661321 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.661656 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.661752 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.664695 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:34.664699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:34.664703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:34.664735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:34.666391 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:34.667002 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:34.667006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:34.667008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:34.667105 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:34.667375 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:34.667378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:34.667380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:34.667595 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:34.667723 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.667956 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.668092 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:27:34.671240 24880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:34.671710 24880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:34.676884 25157 tablet_service.cc:1467] Processing CreateTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 (DEFAULT_TABLE table=test-table [id=4370c04229434c2889b5c8256f9714d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.676987 25017 tablet_service.cc:1467] Processing CreateTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 (DEFAULT_TABLE table=test-table [id=4370c04229434c2889b5c8256f9714d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.677199 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebdc13b6a41d448ba24469c38f0abd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.677263 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebdc13b6a41d448ba24469c38f0abd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.679593 25391 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Bootstrap starting.
I20260618 22:27:34.680022 25392 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Bootstrap starting.
I20260618 22:27:34.680457 25391 tablet_bootstrap.cc:654] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.680801 25392 tablet_bootstrap.cc:654] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.681313 25297 tablet_service.cc:1467] Processing CreateTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 (DEFAULT_TABLE table=test-table [id=4370c04229434c2889b5c8256f9714d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.681600 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebdc13b6a41d448ba24469c38f0abd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.683943 25394 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Bootstrap starting.
I20260618 22:27:34.684903 25394 tablet_bootstrap.cc:654] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.684957 25391 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: No bootstrap required, opened a new log
I20260618 22:27:34.685057 25391 ts_tablet_manager.cc:1397] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:27:34.685443 25391 raft_consensus.cc:348] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.685563 25391 raft_consensus.cc:374] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.685592 25391 raft_consensus.cc:729] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.685704 25391 consensus_queue.cc:260] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.685851 25392 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: No bootstrap required, opened a new log
I20260618 22:27:34.685927 25392 ts_tablet_manager.cc:1397] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260618 22:27:34.685956 25391 ts_tablet_manager.cc:1428] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.686308 25392 raft_consensus.cc:348] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.686412 25392 raft_consensus.cc:374] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.686440 25392 raft_consensus.cc:729] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.687034 25392 consensus_queue.cc:260] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.687325 25392 ts_tablet_manager.cc:1428] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.689116 25394 tablet_bootstrap.cc:492] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: No bootstrap required, opened a new log
I20260618 22:27:34.689213 25394 ts_tablet_manager.cc:1397] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 22:27:34.689661 25394 raft_consensus.cc:348] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.689800 25394 raft_consensus.cc:374] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.689831 25394 raft_consensus.cc:729] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.689951 25394 consensus_queue.cc:260] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.690191 25394 ts_tablet_manager.cc:1428] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.887914 25443 raft_consensus.cc:482] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:34.888077 25443 raft_consensus.cc:504] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.888522 25443 leader_election.cc:288] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:34.889042 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebdc13b6a41d448ba24469c38f0abd90" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69" is_pre_election: true
I20260618 22:27:34.889073 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebdc13b6a41d448ba24469c38f0abd90" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" is_pre_election: true
I20260618 22:27:34.889194 25317 raft_consensus.cc:2454] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:34.889243 25037 raft_consensus.cc:2454] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:34.889510 25108 leader_election.cc:302] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 35b3c11256044769999eb2b190aebb69, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.889696 25443 raft_consensus.cc:2781] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:34.889760 25443 raft_consensus.cc:482] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:34.889796 25443 raft_consensus.cc:3037] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.890908 25443 raft_consensus.cc:504] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.891237 25443 leader_election.cc:288] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:34.891597 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebdc13b6a41d448ba24469c38f0abd90" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
I20260618 22:27:34.891625 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebdc13b6a41d448ba24469c38f0abd90" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69"
I20260618 22:27:34.891729 25037 raft_consensus.cc:3037] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.891732 25317 raft_consensus.cc:3037] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:34.892855 25037 raft_consensus.cc:2454] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:34.892855 25317 raft_consensus.cc:2454] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:34.893185 25108 leader_election.cc:302] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:34.893381 25443 raft_consensus.cc:2781] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:34.893487 25443 raft_consensus.cc:686] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 1 LEADER]: Becoming Leader. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Running, Role: LEADER
I20260618 22:27:34.893625 25443 consensus_queue.cc:237] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.895340 24880 catalog_manager.cc:5495] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:34.929509 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:34.926654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.16.243.130:40469)
0618 22:27:34.926789 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:34.926794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:34.926809 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:34.926876 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:34.926879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:34.926924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:34.926982 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:34.927217 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.927732 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.928545 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:34.928762 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:34.928850 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:34.929085 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:34.929157 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:34.929212 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:34.929319 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:27:34.929372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:27:34.931795 25317 raft_consensus.cc:1261] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.931797 25037 raft_consensus.cc:1261] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:34.932286 25436 consensus_queue.cc:1035] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.932437 25443 consensus_queue.cc:1035] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:34.941476 24879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.941612 24879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:34.943490 24879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Sending DeleteTablet for 3 replicas of tablet ebdc13b6a41d448ba24469c38f0abd90
I20260618 22:27:34.944011 25017 tablet_service.cc:1514] Processing DeleteTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56984
I20260618 22:27:34.944207 25417 tablet_replica.cc:330] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: stopping tablet replica
I20260618 22:27:34.944185 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260618 22:27:34.944330 25417 raft_consensus.cc:2229] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:34.944362 25416 tablet_replica.cc:330] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: stopping tablet replica
I20260618 22:27:34.944465 25417 raft_consensus.cc:2258] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.944465 25416 raft_consensus.cc:2229] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:34.944582 25416 raft_consensus.cc:2258] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.945106 25157 tablet_service.cc:1514] Processing DeleteTablet for tablet ebdc13b6a41d448ba24469c38f0abd90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:34 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56996
I20260618 22:27:34.945320 25418 tablet_replica.cc:330] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: stopping tablet replica
I20260618 22:27:34.945422 25418 raft_consensus.cc:2229] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:34.945528 24879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:27:34.945694 25418 raft_consensus.cc:2258] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:34.945818 25417 ts_tablet_manager.cc:1905] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:27:34.945931 24879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:34.946489 25418 ts_tablet_manager.cc:1905] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.948367 25417 ts_tablet_manager.cc:1918] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.948434 25417 log.cc:1196] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/ebdc13b6a41d448ba24469c38f0abd90
I20260618 22:27:34.948680 25417 ts_tablet_manager.cc:1939] T ebdc13b6a41d448ba24469c38f0abd90 P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Deleting consensus metadata
I20260618 22:27:34.948762 25418 ts_tablet_manager.cc:1918] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.948838 25418 log.cc:1196] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ebdc13b6a41d448ba24469c38f0abd90
I20260618 22:27:34.949054 25418 ts_tablet_manager.cc:1939] T ebdc13b6a41d448ba24469c38f0abd90 P 26f58d3c6d334f2da5446be424f89d33: Deleting consensus metadata
I20260618 22:27:34.949422 24864 catalog_manager.cc:4841] TS 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961): tablet ebdc13b6a41d448ba24469c38f0abd90 (table test-table [id=4370c04229434c2889b5c8256f9714d4]) successfully deleted
I20260618 22:27:34.949688 24867 catalog_manager.cc:4841] TS 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469): tablet ebdc13b6a41d448ba24469c38f0abd90 (table test-table [id=4370c04229434c2889b5c8256f9714d4]) successfully deleted
I20260618 22:27:34.950377 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 49121085be274298bc5ba590a86cdd8b (DEFAULT_TABLE table=test-table [id=463c964892ad4375a904cc675449a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.950659 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49121085be274298bc5ba590a86cdd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.950691 25297 tablet_service.cc:1467] Processing CreateTablet for tablet 49121085be274298bc5ba590a86cdd8b (DEFAULT_TABLE table=test-table [id=463c964892ad4375a904cc675449a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.950915 25297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49121085be274298bc5ba590a86cdd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.952754 25392 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Bootstrap starting.
I20260618 22:27:34.953159 25394 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Bootstrap starting.
I20260618 22:27:34.953471 25416 ts_tablet_manager.cc:1905] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:34.953559 25392 tablet_bootstrap.cc:654] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.953928 25394 tablet_bootstrap.cc:654] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.954851 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 49121085be274298bc5ba590a86cdd8b (DEFAULT_TABLE table=test-table [id=463c964892ad4375a904cc675449a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:34.954974 25394 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: No bootstrap required, opened a new log
I20260618 22:27:34.955084 25394 ts_tablet_manager.cc:1397] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:34.955093 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49121085be274298bc5ba590a86cdd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:34.955479 25394 raft_consensus.cc:348] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.955564 25416 ts_tablet_manager.cc:1918] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:34.955580 25394 raft_consensus.cc:374] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.955610 25394 raft_consensus.cc:729] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3c11256044769999eb2b190aebb69, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.955747 25394 consensus_queue.cc:260] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.955613 25416 log.cc:1196] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/ebdc13b6a41d448ba24469c38f0abd90
I20260618 22:27:34.956035 25394 ts_tablet_manager.cc:1428] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:34.956079 25416 ts_tablet_manager.cc:1939] T ebdc13b6a41d448ba24469c38f0abd90 P 35b3c11256044769999eb2b190aebb69: Deleting consensus metadata
I20260618 22:27:34.956741 24867 catalog_manager.cc:4841] TS 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885): tablet ebdc13b6a41d448ba24469c38f0abd90 (table test-table [id=4370c04229434c2889b5c8256f9714d4]) successfully deleted
I20260618 22:27:34.957299 25391 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Bootstrap starting.
I20260618 22:27:34.958137 25391 tablet_bootstrap.cc:654] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:34.958734 25392 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: No bootstrap required, opened a new log
I20260618 22:27:34.958833 25392 ts_tablet_manager.cc:1397] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 22:27:34.959224 25392 raft_consensus.cc:348] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.959332 25392 raft_consensus.cc:374] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.959362 25392 raft_consensus.cc:729] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddf01d4c1c942aeaaaaea8cec9c63cd, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.959478 25392 consensus_queue.cc:260] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.959723 25392 ts_tablet_manager.cc:1428] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:27:34.959738 25391 tablet_bootstrap.cc:492] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: No bootstrap required, opened a new log
I20260618 22:27:34.959816 25391 ts_tablet_manager.cc:1397] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 22:27:34.960191 25391 raft_consensus.cc:348] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.960302 25391 raft_consensus.cc:374] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:34.960333 25391 raft_consensus.cc:729] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Initialized, Role: FOLLOWER
I20260618 22:27:34.960462 25391 consensus_queue.cc:260] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:34.960731 25391 ts_tablet_manager.cc:1428] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:27:35.008419 25443 raft_consensus.cc:482] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:35.008562 25443 raft_consensus.cc:504] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:35.008996 25443 leader_election.cc:288] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:35.009442 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49121085be274298bc5ba590a86cdd8b" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" is_pre_election: true
I20260618 22:27:35.009446 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49121085be274298bc5ba590a86cdd8b" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69" is_pre_election: true
I20260618 22:27:35.009595 25037 raft_consensus.cc:2454] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:35.009595 25317 raft_consensus.cc:2454] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 0.
I20260618 22:27:35.009899 25108 leader_election.cc:302] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 8ddf01d4c1c942aeaaaaea8cec9c63cd; no voters:
I20260618 22:27:35.010071 25443 raft_consensus.cc:2781] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:35.010147 25443 raft_consensus.cc:482] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:35.010191 25443 raft_consensus.cc:3037] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:35.011471 25443 raft_consensus.cc:504] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:35.011855 25443 leader_election.cc:288] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961), 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885)
I20260618 22:27:35.012214 25037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49121085be274298bc5ba590a86cdd8b" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd"
I20260618 22:27:35.012243 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49121085be274298bc5ba590a86cdd8b" candidate_uuid: "26f58d3c6d334f2da5446be424f89d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3c11256044769999eb2b190aebb69"
I20260618 22:27:35.012341 25037 raft_consensus.cc:3037] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:35.012382 25317 raft_consensus.cc:3037] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:35.013243 25317 raft_consensus.cc:2454] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:35.013429 25037 raft_consensus.cc:2454] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f58d3c6d334f2da5446be424f89d33 in term 1.
I20260618 22:27:35.013510 25111 leader_election.cc:302] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f58d3c6d334f2da5446be424f89d33, 35b3c11256044769999eb2b190aebb69; no voters:
I20260618 22:27:35.013674 25443 raft_consensus.cc:2781] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:35.013764 25443 raft_consensus.cc:686] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 1 LEADER]: Becoming Leader. State: Replica: 26f58d3c6d334f2da5446be424f89d33, State: Running, Role: LEADER
I20260618 22:27:35.013902 25443 consensus_queue.cc:237] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } }
I20260618 22:27:35.015594 24879 catalog_manager.cc:5495] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "26f58d3c6d334f2da5446be424f89d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f58d3c6d334f2da5446be424f89d33" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:35.041919 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:35.038471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:40469 (local address 127.16.243.130:40445)
0618 22:27:35.038616 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:35.038631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:35.038702 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:35.038934 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:35.038936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:35.038941 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:27:35.039019 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:35.039023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:35.040003 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:35.040006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:35.040623 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:35.040632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:35.041201 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:35.041205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:35.041273 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:35.041303 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:27:35.041607 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:27:35.041612 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:35.041675 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:27:35.041773 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260618 22:27:35.041936 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:35.038567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:40445 (local address 127.16.243.130:40469)
0618 22:27:35.038681 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:35.038686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:35.038700 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:35.038798 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:35.038801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:35.038850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:35.038924 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:35.039098 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:35.039866 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:35.040752 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:35.041075 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:35.041186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:35.041425 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:35.041512 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:35.041588 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:35.041699 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 22:27:35.041793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":46}
I20260618 22:27:35.048120 25317 raft_consensus.cc:1261] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:35.048198 25036 raft_consensus.cc:1261] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Refusing update from remote peer 26f58d3c6d334f2da5446be424f89d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:35.048570 25397 consensus_queue.cc:1035] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b3c11256044769999eb2b190aebb69" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:35.048856 25397 consensus_queue.cc:1035] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddf01d4c1c942aeaaaaea8cec9c63cd" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:35.085902 24879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:35.086057 24879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:35.088074 24879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9157c2c7030641f08f6e297e1fbbb237: Sending DeleteTablet for 3 replicas of tablet 49121085be274298bc5ba590a86cdd8b
I20260618 22:27:35.088667 25017 tablet_service.cc:1514] Processing DeleteTablet for tablet 49121085be274298bc5ba590a86cdd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:35 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56984
I20260618 22:27:35.088721 25297 tablet_service.cc:1514] Processing DeleteTablet for tablet 49121085be274298bc5ba590a86cdd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:35 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260618 22:27:35.088863 25417 tablet_replica.cc:330] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd: stopping tablet replica
I20260618 22:27:35.088925 25416 tablet_replica.cc:330] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: stopping tablet replica
I20260618 22:27:35.088990 25417 raft_consensus.cc:2229] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:35.089020 25416 raft_consensus.cc:2229] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:35.089099 25417 raft_consensus.cc:2258] T 49121085be274298bc5ba590a86cdd8b P 8ddf01d4c1c942aeaaaaea8cec9c63cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:35.089128 25416 raft_consensus.cc:2258] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:35.089430 25157 tablet_service.cc:1514] Processing DeleteTablet for tablet 49121085be274298bc5ba590a86cdd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:35 UTC) from {username='oryx', principal='oryx/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56996
I20260618 22:27:35.089591 25418 tablet_replica.cc:330] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: stopping tablet replica
I20260618 22:27:35.089687 25418 raft_consensus.cc:2229] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:35.089793 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24952
I20260618 22:27:35.089880 25418 raft_consensus.cc:2258] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:35.090194 25416 ts_tablet_manager.cc:1905] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:35.092779 25416 ts_tablet_manager.cc:1918] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:35.092849 25416 log.cc:1196] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/49121085be274298bc5ba590a86cdd8b
I20260618 22:27:35.093091 25416 ts_tablet_manager.cc:1939] T 49121085be274298bc5ba590a86cdd8b P 35b3c11256044769999eb2b190aebb69: Deleting consensus metadata
I20260618 22:27:35.093088 25418 ts_tablet_manager.cc:1905] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:35.093742 24867 catalog_manager.cc:4841] TS 35b3c11256044769999eb2b190aebb69 (127.16.243.131:35885): tablet 49121085be274298bc5ba590a86cdd8b (table test-table [id=463c964892ad4375a904cc675449a999]) successfully deleted
I20260618 22:27:35.095489 25418 ts_tablet_manager.cc:1918] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:27:35.095567 25418 log.cc:1196] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/49121085be274298bc5ba590a86cdd8b
I20260618 22:27:35.095794 25418 ts_tablet_manager.cc:1939] T 49121085be274298bc5ba590a86cdd8b P 26f58d3c6d334f2da5446be424f89d33: Deleting consensus metadata
I20260618 22:27:35.096513 24867 catalog_manager.cc:4841] TS 26f58d3c6d334f2da5446be424f89d33 (127.16.243.130:40469): tablet 49121085be274298bc5ba590a86cdd8b (table test-table [id=463c964892ad4375a904cc675449a999]) successfully deleted
W20260618 22:27:35.098606 24864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:44961 (error 108)
I20260618 22:27:35.099393 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25092
I20260618 22:27:35.100490 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:35.099305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44961 (local address 127.0.0.1:57012)
0618 22:27:35.100333 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:35.100410 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":47,"threads_started":1}
W20260618 22:27:35.100594 24864 catalog_manager.cc:4584] TS 8ddf01d4c1c942aeaaaaea8cec9c63cd (127.16.243.129:44961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49121085be274298bc5ba590a86cdd8b: Network error: Client connection negotiation failed: client connection to 127.16.243.129:44961: connect: Connection refused (error 111)
I20260618 22:27:35.107383 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25232
I20260618 22:27:35.112932 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 24848
2026-06-18T22:27:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3267 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:35 dist-test-slave-jm5g krb5kdc[25485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:35 dist-test-slave-jm5g krb5kdc[25485](info): set up 2 sockets
Jun 18 22:27:35 dist-test-slave-jm5g krb5kdc[25485](info): commencing operation
krb5kdc: starting...
W20260618 22:27:36.070960 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:36 dist-test-slave-jm5g krb5kdc[25485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:36.189149 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:32871
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:32871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:36.300485 25498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:36.300794 25498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:36.300851 25498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:36.305291 25498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:36.305375 25498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:36.305406 25498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:36.305428 25498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:36.305450 25498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:36.311422 25498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:32871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:32871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:36.312572 25498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:36.313627 25498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:36.322080 25503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:36.322080 25506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:36.322118 25504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:36.322341 25498 server_base.cc:1034] running on GCE node
I20260618 22:27:36.323097 25498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:36.324301 25498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:36.325462 25498 hybrid_clock.cc:648] HybridClock initialized: now 1781821656325435 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-18T22:27:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1224 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:36 dist-test-slave-jm5g krb5kdc[25511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:36 dist-test-slave-jm5g krb5kdc[25511](info): set up 2 sockets
Jun 18 22:27:36 dist-test-slave-jm5g krb5kdc[25511](info): commencing operation
krb5kdc: starting...
W20260618 22:27:37.290966 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:37 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:37.405169 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42745
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:37.513605 25524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:37.513897 25524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:37.513938 25524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:37.518010 25524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:37.518090 25524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:37.518108 25524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:37.518124 25524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:37.518138 25524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:37.523703 25524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:37.524864 25524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:37.525879 25524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:37.534278 25530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:37.534296 25529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:37.534415 25524 server_base.cc:1034] running on GCE node
W20260618 22:27:37.534281 25532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:37.535171 25524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:37.536340 25524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:37.537503 25524 hybrid_clock.cc:648] HybridClock initialized: now 1781821657537446 us; error 74 us; skew 500 ppm
Jun 18 22:27:37 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821657, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:37.551450 25524 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:37.552484 25524 webserver.cc:460] Webserver started at http://127.16.243.190:34189/ using document root <none> and password file <none>
I20260618 22:27:37.553053 25524 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:37.553123 25524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:37.553331 25524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:37.555528 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "8476a427cd2f40f1a13a62fdc74dfb7c"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "d1c92335664a3d2f80887ca066a179be"
server_key_iv: "7d5fcae9717f2bf7dbd3a2e6d236c1e9"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.556078 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "8476a427cd2f40f1a13a62fdc74dfb7c"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "d1c92335664a3d2f80887ca066a179be"
server_key_iv: "7d5fcae9717f2bf7dbd3a2e6d236c1e9"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.559902 25524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:37.562350 25539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:37.563702 25524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:37.563829 25524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "8476a427cd2f40f1a13a62fdc74dfb7c"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "d1c92335664a3d2f80887ca066a179be"
server_key_iv: "7d5fcae9717f2bf7dbd3a2e6d236c1e9"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.563954 25524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:37.573526 25524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:37.576551 25524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:37.576735 25524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:37.583714 25524 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:42745
I20260618 22:27:37.583719 25591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:42745 every 8 connection(s)
I20260618 22:27:37.584749 25524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:37.587568 25592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:37.591418 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25524
I20260618 22:27:37.591547 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:37.591825 17358 external_mini_cluster.cc:1428] Setting key fbe3091f4c601705aaa2568a4c8b5394
I20260618 22:27:37.593477 25592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Bootstrap starting.
I20260618 22:27:37.595923 25592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:37.596627 25592 log.cc:824] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:37.598768 25592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: No bootstrap required, opened a new log
Jun 18 22:27:37 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:37.602278 25592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:37.602522 25592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:37.602598 25592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8476a427cd2f40f1a13a62fdc74dfb7c, State: Initialized, Role: FOLLOWER
I20260618 22:27:37.603117 25592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:37.603324 25597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:37.603488 25597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:37.603338 25592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:37.603828 25592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:37.603899 25592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:37.603809 25597 leader_election.cc:302] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8476a427cd2f40f1a13a62fdc74dfb7c; no voters:
I20260618 22:27:37.604213 25597 leader_election.cc:288] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260618 22:27:37.604926 25592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:37.605057 25592 leader_election.cc:302] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8476a427cd2f40f1a13a62fdc74dfb7c; no voters:
I20260618 22:27:37.605118 25598 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260618 22:27:37.605196 25592 leader_election.cc:288] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:37.605649 25597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:37.605645 25599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } } }
I20260618 22:27:37.605778 25599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:27:37.605906 25597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 LEADER]: Becoming Leader. State: Replica: 8476a427cd2f40f1a13a62fdc74dfb7c, State: Running, Role: LEADER
I20260618 22:27:37.606191 25597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:37.606472 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:37.593194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.16.243.190:42745)
0618 22:27:37.593624 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:37.593643 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:37.593678 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:27:37.594314 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:37.594331 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:37.594723 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:37.595011 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:37.595826 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.596868 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.597912 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.598395 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.598514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:37.601211 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:37.601216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:37.601230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:37.601257 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:37.603275 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:37.603791 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:37.603795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:37.603799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:37.603983 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:37.604256 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:37.604259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:37.604261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:37.604689 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:37.604837 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:37.605163 (+ 326us) server_negotiation.cc:299] Negotiation successful
0618 22:27:37.605584 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260618 22:27:37.606527 25592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:37.608071 25598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8476a427cd2f40f1a13a62fdc74dfb7c. Latest consensus state: current_term: 1 leader_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } } }
I20260618 22:27:37.608181 25598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:37.609066 25604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:37.611910 25604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:37.617763 25604 catalog_manager.cc:1347] Generated new cluster ID: f57e95f2c3f543fe981514f58cfd3f82
I20260618 22:27:37.617861 25604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:37.652642 25604 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:37.653666 25604 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:37.661556 25604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Generated new TSK 0
I20260618 22:27:37.662429 25604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:37.699389 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:37.808549 25618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:37.808872 25618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:37.808919 25618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:37.813032 25618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:37.813103 25618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:37.813161 25618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:37.818540 25618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:37.819653 25618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:37.820667 25618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:37.830437 25624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:37.830437 25626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:37.830461 25623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:37.830765 25618 server_base.cc:1034] running on GCE node
I20260618 22:27:37.831187 25618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:37.831871 25618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:37.833043 25618 hybrid_clock.cc:648] HybridClock initialized: now 1781821657833020 us; error 38 us; skew 500 ppm
Jun 18 22:27:37 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821657, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:37.835907 25618 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:37.836807 25618 webserver.cc:460] Webserver started at http://127.16.243.129:39797/ using document root <none> and password file <none>
I20260618 22:27:37.837334 25618 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:37.837395 25618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:37.837574 25618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:37.839706 25618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "9cd16aec5aa24ae4bb06e7fc48c58cdf"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "bf1e48bf0952f300c0207f0ff07c6c44"
server_key_iv: "f5ba20efef4be173e3f889b88cbb0ab2"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.840238 25618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd16aec5aa24ae4bb06e7fc48c58cdf"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "bf1e48bf0952f300c0207f0ff07c6c44"
server_key_iv: "f5ba20efef4be173e3f889b88cbb0ab2"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.844090 25618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:37.846447 25633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:37.847689 25618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:37.847803 25618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "9cd16aec5aa24ae4bb06e7fc48c58cdf"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "bf1e48bf0952f300c0207f0ff07c6c44"
server_key_iv: "f5ba20efef4be173e3f889b88cbb0ab2"
server_key_version: "encryptionkey@0"
I20260618 22:27:37.847915 25618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:37.864557 25618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:37.867558 25618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:37.867734 25618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:37.868340 25618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:37.869155 25618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:37.869199 25618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:37.869252 25618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:37.869282 25618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:37.877897 25618 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:35341
I20260618 22:27:37.877921 25746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:35341 every 8 connection(s)
I20260618 22:27:37.878825 25618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:37.885475 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25618
I20260618 22:27:37.885592 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:37.885891 17358 external_mini_cluster.cc:1428] Setting key 953462952378d92aea0a5525da56466e
Jun 18 22:27:37 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821657, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:37.891865 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:37.880579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:54271 (local address 127.16.243.190:42745)
0618 22:27:37.880713 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:37.880717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:37.881741 (+ 1024us) server_negotiation.cc:407] Connection header received
0618 22:27:37.882685 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:37.882689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:37.882748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:37.882818 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:37.884029 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.884565 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.885694 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.886030 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.886158 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:37.888476 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:37.888480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:37.888483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:37.888512 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:37.889826 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:37.890310 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:37.890321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:37.890322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:37.890397 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:37.890780 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:37.890783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:37.890785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:37.890967 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:37.891052 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:37.891566 (+ 514us) server_negotiation.cc:299] Negotiation successful
0618 22:27:37.891697 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:27:37.892711 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:37.880960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:54271)
0618 22:27:37.881589 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:37.881631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:37.882441 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:37.883006 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:37.883014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:37.883377 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:37.883863 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:37.883890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.884683 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.884686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:37.885572 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:37.885582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:37.886190 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:37.886192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:37.886357 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:37.886936 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:37.886954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:37.888311 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:37.889937 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:37.889943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:37.889959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:37.890214 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:37.890480 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:37.890483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:37.890485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:37.890694 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:37.891068 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:37.891075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:37.891328 (+ 253us) client_negotiation.cc:769] Sending connection context
0618 22:27:37.891494 (+ 166us) client_negotiation.cc:240] Negotiation successful
0618 22:27:37.891849 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:27:37.893997 25747 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42745
I20260618 22:27:37.894243 25747 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:37.894738 25747 heartbeater.cc:511] Master 127.16.243.190:42745 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:27:37.896478 25556 ts_manager.cc:194] Registered new tserver with Master: 9cd16aec5aa24ae4bb06e7fc48c58cdf (127.16.243.129:35341)
I20260618 22:27:37.897642 25556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:54271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:37.910969 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:38.019491 25752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:38.019785 25752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:38.019847 25752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:38.023931 25752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:38.024006 25752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:38.024091 25752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:38.029595 25752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:38.030697 25752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:38.031706 25752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:38.040973 25760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.041062 25757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.041096 25758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.041358 25752 server_base.cc:1034] running on GCE node
I20260618 22:27:38.041725 25752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:38.042368 25752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:38.043543 25752 hybrid_clock.cc:648] HybridClock initialized: now 1781821658043532 us; error 46 us; skew 500 ppm
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:38.046336 25752 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:38.047379 25752 webserver.cc:460] Webserver started at http://127.16.243.130:41323/ using document root <none> and password file <none>
I20260618 22:27:38.047931 25752 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:38.048004 25752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:38.048225 25752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:38.050364 25752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "cb3f9ffeec1046d9b2a34de4582a359f"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "45e64f903900a0cbb29b005b83a197a9"
server_key_iv: "582b07f7c1b7a75da6e142921cb3b4eb"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.050968 25752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "cb3f9ffeec1046d9b2a34de4582a359f"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "45e64f903900a0cbb29b005b83a197a9"
server_key_iv: "582b07f7c1b7a75da6e142921cb3b4eb"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.054803 25752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:38.057416 25767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.058858 25752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:38.058972 25752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "cb3f9ffeec1046d9b2a34de4582a359f"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "45e64f903900a0cbb29b005b83a197a9"
server_key_iv: "582b07f7c1b7a75da6e142921cb3b4eb"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.059100 25752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:38.080302 25752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:38.083642 25752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:38.083817 25752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:38.084432 25752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:38.085260 25752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:38.085306 25752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.085346 25752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:38.085374 25752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.093845 25752 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:33375
I20260618 22:27:38.093868 25880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:33375 every 8 connection(s)
I20260618 22:27:38.094782 25752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:38.097229 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25752
I20260618 22:27:38.097371 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:38.097663 17358 external_mini_cluster.cc:1428] Setting key 6fcc65ba132a8ae198b12a71a98bbd83
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:38.110008 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.096539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57433 (local address 127.16.243.190:42745)
0618 22:27:38.096668 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:38.096672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:38.097824 (+ 1152us) server_negotiation.cc:407] Connection header received
0618 22:27:38.098871 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:38.098874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:38.098919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:38.098998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:38.100509 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.101055 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.102711 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.102950 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.103033 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.106053 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:38.106057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:38.106060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:38.106089 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:38.107681 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:38.108225 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:38.108229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:38.108231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:38.108341 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:38.108704 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:38.108708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:38.108709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:38.108916 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:38.109012 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:38.109660 (+ 648us) server_negotiation.cc:299] Negotiation successful
0618 22:27:38.109804 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:38.110932 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.096929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:57433)
0618 22:27:38.097660 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:38.097719 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:38.098676 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:38.099187 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:38.099198 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:38.099691 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:38.100363 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.100381 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.101179 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.101183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.102569 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.102581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.103105 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.103109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.103341 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.104036 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:38.104059 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:38.105857 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:38.107801 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:38.107808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:38.107828 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:38.108109 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:38.108438 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:38.108441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:38.108444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:38.108601 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:38.109020 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:38.109027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:38.109399 (+ 372us) client_negotiation.cc:769] Sending connection context
0618 22:27:38.109591 (+ 192us) client_negotiation.cc:240] Negotiation successful
0618 22:27:38.109994 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":106,"threads_started":1}
I20260618 22:27:38.112329 25881 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42745
I20260618 22:27:38.112612 25881 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:38.113204 25881 heartbeater.cc:511] Master 127.16.243.190:42745 requested a full tablet report, sending...
I20260618 22:27:38.114289 25556 ts_manager.cc:194] Registered new tserver with Master: cb3f9ffeec1046d9b2a34de4582a359f (127.16.243.130:33375)
I20260618 22:27:38.114915 25556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:57433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:38.125336 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:38.235877 25886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:38.236186 25886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:38.236248 25886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:38.240553 25886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:38.240622 25886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:38.240712 25886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:38.246237 25886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:38.247408 25886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:38.248425 25886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:38.258111 25891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.258230 25886 server_base.cc:1034] running on GCE node
W20260618 22:27:38.258109 25894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.258112 25892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.258942 25886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:38.259611 25886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:38.260803 25886 hybrid_clock.cc:648] HybridClock initialized: now 1781821658260778 us; error 36 us; skew 500 ppm
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:38.263674 25886 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:38.264613 25886 webserver.cc:460] Webserver started at http://127.16.243.131:40531/ using document root <none> and password file <none>
I20260618 22:27:38.265151 25886 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:38.265213 25886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:38.265416 25886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:38.267557 25886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "08d3c39c27464fcc9863757a878e1c1b"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "18a9ea1542539fa0025b9a223aabbfca"
server_key_iv: "fa2d6e7dea5014918d398616cb1c5448"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.268128 25886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "08d3c39c27464fcc9863757a878e1c1b"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "18a9ea1542539fa0025b9a223aabbfca"
server_key_iv: "fa2d6e7dea5014918d398616cb1c5448"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.271885 25886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:38.274355 25901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.275629 25886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:38.275749 25886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "08d3c39c27464fcc9863757a878e1c1b"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "18a9ea1542539fa0025b9a223aabbfca"
server_key_iv: "fa2d6e7dea5014918d398616cb1c5448"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.275859 25886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:38.323123 25886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:38.326315 25886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:38.326485 25886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:38.327095 25886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:38.327915 25886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:38.327960 25886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.328001 25886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:38.328029 25886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.336447 25886 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:41519
I20260618 22:27:38.336521 26014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:41519 every 8 connection(s)
I20260618 22:27:38.337356 25886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:38.341907 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 25886
I20260618 22:27:38.342042 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:38.342321 17358 external_mini_cluster.cc:1428] Setting key 3283c03f6879b58a2871b008108195e0
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:38.350898 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.339128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:41463 (local address 127.16.243.190:42745)
0618 22:27:38.339268 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:38.339272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:38.340107 (+ 835us) server_negotiation.cc:407] Connection header received
0618 22:27:38.341337 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:38.341340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:38.341389 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:38.341465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:38.342995 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.343612 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.344902 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.345113 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.345193 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.347684 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:38.347687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:38.347689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:38.347717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:38.349011 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:38.349478 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:38.349481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:38.349482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:38.349551 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:38.349811 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:38.349813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:38.349814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:38.349980 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:38.350053 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:38.350632 (+ 579us) server_negotiation.cc:299] Negotiation successful
0618 22:27:38.350747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:27:38.351665 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.339518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:41463)
0618 22:27:38.339944 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:38.339988 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:38.341111 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:38.341653 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:38.341662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:38.342154 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:38.342834 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.342856 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.343734 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.343738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.344793 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.344803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.345236 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.345238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.345405 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.346006 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:38.346026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:38.347519 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:38.349118 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:38.349123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:38.349136 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:38.349403 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:38.349615 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:38.349617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:38.349618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:38.349749 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:38.350055 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:38.350060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:38.350310 (+ 250us) client_negotiation.cc:769] Sending connection context
0618 22:27:38.350497 (+ 187us) client_negotiation.cc:240] Negotiation successful
0618 22:27:38.350829 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260618 22:27:38.352803 26015 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42745
I20260618 22:27:38.353065 26015 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:38.353545 26015 heartbeater.cc:511] Master 127.16.243.190:42745 requested a full tablet report, sending...
I20260618 22:27:38.354619 25556 ts_manager.cc:194] Registered new tserver with Master: 08d3c39c27464fcc9863757a878e1c1b (127.16.243.131:41519)
I20260618 22:27:38.355230 25556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:41463
I20260618 22:27:38.356216 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
Entry for principal oryx/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab.
WARNING: no policy specified for oryx/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
Entry for principal oryx/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab.
WARNING: no policy specified for oryx/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
Entry for principal oryx/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab.
I20260618 22:27:38.421291 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25618
I20260618 22:27:38.425633 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25752
I20260618 22:27:38.429800 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25886
I20260618 22:27:38.433697 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 25524
I20260618 22:27:38.438458 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42745
--webserver_interface=127.16.243.190
--webserver_port=34189
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:38.552309 26024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:38.552587 26024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:38.552630 26024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:38.556722 26024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:38.556799 26024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:38.556838 26024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:38.556867 26024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:38.556896 26024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:38.562325 26024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:38.563408 26024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:38.564426 26024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:38.572831 26032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.572804 26030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.572798 26029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.573150 26024 server_base.cc:1034] running on GCE node
I20260618 22:27:38.573701 26024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:38.574856 26024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:38.576011 26024 hybrid_clock.cc:648] HybridClock initialized: now 1781821658575991 us; error 35 us; skew 500 ppm
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:38.578673 26024 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:38.579596 26024 webserver.cc:460] Webserver started at http://127.16.243.190:34189/ using document root <none> and password file <none>
I20260618 22:27:38.580147 26024 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:38.580226 26024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:38.584177 26024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:38.586233 26039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.587527 26024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:38.587651 26024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "8476a427cd2f40f1a13a62fdc74dfb7c"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "d1c92335664a3d2f80887ca066a179be"
server_key_iv: "7d5fcae9717f2bf7dbd3a2e6d236c1e9"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.588224 26024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:38.612133 26024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:38.615216 26024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:38.615391 26024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:38.622471 26024 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:42745
I20260618 22:27:38.622493 26091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:42745 every 8 connection(s)
I20260618 22:27:38.623688 26024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:38.627871 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26024
I20260618 22:27:38.628240 26092 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:27:38.628284 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:35341
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:42745
--webserver_port=39797
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:27:38.629932 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Bootstrap starting.
I20260618 22:27:38.649201 26092 log.cc:824] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:38.657928 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:27:38.658385 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Bootstrap complete.
I20260618 22:27:38.663630 26092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:38.664409 26092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8476a427cd2f40f1a13a62fdc74dfb7c, State: Initialized, Role: FOLLOWER
I20260618 22:27:38.665050 26092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:38.665223 26092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:38.665293 26092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:38.665397 26092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:27:38.667881 26092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:38.668269 26092 leader_election.cc:302] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8476a427cd2f40f1a13a62fdc74dfb7c; no voters:
I20260618 22:27:38.668651 26092 leader_election.cc:288] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:27:38.668815 26096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:27:38.669068 26096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [term 2 LEADER]: Becoming Leader. State: Replica: 8476a427cd2f40f1a13a62fdc74dfb7c, State: Running, Role: LEADER
I20260618 22:27:38.669436 26096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } }
I20260618 22:27:38.669958 26092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:38.670533 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8476a427cd2f40f1a13a62fdc74dfb7c. Latest consensus state: current_term: 2 leader_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } } }
I20260618 22:27:38.671257 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:38.671146 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476a427cd2f40f1a13a62fdc74dfb7c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42745 } } }
I20260618 22:27:38.671998 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:38.672080 26105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:38.675421 26105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:38.676079 26105 catalog_manager.cc:1259] Loaded cluster ID: f57e95f2c3f543fe981514f58cfd3f82
I20260618 22:27:38.676134 26105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:38.678395 26105 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:38.679111 26105 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8476a427cd2f40f1a13a62fdc74dfb7c: Loaded TSK: 0
I20260618 22:27:38.680336 26105 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:27:38.754691 26094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:38.754974 26094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:38.755036 26094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:38.759156 26094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:38.759227 26094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:38.759347 26094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:38.764873 26094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.129.keytab
--principal=oryx/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=39797
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:38.765949 26094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:38.767057 26094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:38.776407 26119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.776407 26121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.776470 26118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.777354 26094 server_base.cc:1034] running on GCE node
I20260618 22:27:38.777765 26094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:38.778432 26094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:38.779606 26094 hybrid_clock.cc:648] HybridClock initialized: now 1781821658779571 us; error 47 us; skew 500 ppm
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:38.782660 26094 init.cc:377] Logged in from keytab as oryx/127.16.243.129@KRBTEST.COM (short username oryx)
I20260618 22:27:38.783565 26094 webserver.cc:460] Webserver started at http://127.16.243.129:39797/ using document root <none> and password file <none>
I20260618 22:27:38.784166 26094 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:38.784251 26094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:38.788336 26094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:27:38.790660 26128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.792038 26094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:27:38.792205 26094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "9cd16aec5aa24ae4bb06e7fc48c58cdf"
format_stamp: "Formatted at 2026-06-18 22:27:37 on dist-test-slave-jm5g"
server_key: "bf1e48bf0952f300c0207f0ff07c6c44"
server_key_iv: "f5ba20efef4be173e3f889b88cbb0ab2"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.792868 26094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:38.809120 26094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:38.812352 26094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:38.812533 26094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:38.813122 26094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:38.813941 26094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:38.813987 26094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.814038 26094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:38.814066 26094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.822377 26094 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:35341
I20260618 22:27:38.822435 26241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:35341 every 8 connection(s)
I20260618 22:27:38.823437 26094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:38.824453 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26094
I20260618 22:27:38.824868 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:33375
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:42745
--webserver_port=41323
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:38.837847 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.825856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:53751)
0618 22:27:38.826685 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:38.826745 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:38.827718 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:38.829053 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:38.829060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:38.829408 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:38.829881 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.829895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.831856 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.831859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.832897 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.832908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.833975 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.833978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.834121 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.834728 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:38.834746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:38.837009 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":122,"threads_started":1}
W20260618 22:27:38.838201 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:38.826789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:53751 (local address 127.16.243.190:42745)
0618 22:27:38.827185 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:38.827200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:38.827237 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:27:38.828177 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:38.828192 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:38.828599 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:38.829002 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:38.830052 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.831646 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.833029 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.833809 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.833988 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.837286 (+ 3298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:53751: BlockingRecv error: recv got EOF from 127.16.243.129:53751 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
W20260618 22:27:38.838508 26242 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:42745 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.243.190:42745: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:38.843979 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.839001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:43725)
0618 22:27:38.839101 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:38.839116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:38.839191 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:38.839420 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:38.839422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:38.839552 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:38.839652 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.839657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.840627 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.840628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.841159 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.841165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.841722 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.841724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.841786 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.842226 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:38.842238 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:38.843862 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260618 22:27:38.844153 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:38.839091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:43725 (local address 127.16.243.190:42745)
0618 22:27:38.839202 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:38.839206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:38.839221 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:38.839265 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:38.839268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:38.839314 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:38.839399 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:38.839742 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.840541 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.841263 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.841607 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.841719 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.844041 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:43725: BlockingRecv error: recv got EOF from 127.16.243.129:43725 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:38.849079 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:38.844466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:60151)
0618 22:27:38.844780 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:38.844791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:38.844870 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:38.845079 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:38.845082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:38.845215 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:38.845286 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.845291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.845975 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.845978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.846491 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:38.846499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.846907 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.846910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:38.846980 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.847399 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:38.847410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:38.848921 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
W20260618 22:27:38.849160 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:38.844566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:60151 (local address 127.16.243.190:42745)
0618 22:27:38.844679 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:38.844683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:38.844850 (+ 167us) server_negotiation.cc:407] Connection header received
0618 22:27:38.844948 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:38.844951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:38.844994 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:38.845056 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:38.845383 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.845879 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.846605 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:38.846815 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:38.846905 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:38.849057 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:60151: BlockingRecv error: recv got EOF from 127.16.243.129:60151 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260618 22:27:38.849308 26242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:27:38.937822 26244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:38.938098 26244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:38.938143 26244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:38.942322 26244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:38.942394 26244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:38.942462 26244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:38.947882 26244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.130.keytab
--principal=oryx/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:33375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=41323
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:38.948951 26244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:38.949990 26244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:38.959679 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.959704 26253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:38.959712 26252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:38.959924 26244 server_base.cc:1034] running on GCE node
I20260618 22:27:38.960359 26244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:38.961031 26244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:38.962224 26244 hybrid_clock.cc:648] HybridClock initialized: now 1781821658962209 us; error 51 us; skew 500 ppm
Jun 18 22:27:38 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821658, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:38.964993 26244 init.cc:377] Logged in from keytab as oryx/127.16.243.130@KRBTEST.COM (short username oryx)
I20260618 22:27:38.965888 26244 webserver.cc:460] Webserver started at http://127.16.243.130:41323/ using document root <none> and password file <none>
I20260618 22:27:38.966436 26244 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:38.966511 26244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:38.970466 26244 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:38.972725 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.973984 26244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:27:38.974087 26244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "cb3f9ffeec1046d9b2a34de4582a359f"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "45e64f903900a0cbb29b005b83a197a9"
server_key_iv: "582b07f7c1b7a75da6e142921cb3b4eb"
server_key_version: "encryptionkey@0"
I20260618 22:27:38.974822 26244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:38.993512 26244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:38.996807 26244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:38.997021 26244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:38.997618 26244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:38.998446 26244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:38.998493 26244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:38.998534 26244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:38.998588 26244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:39.006997 26244 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:33375
I20260618 22:27:39.007046 26375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:33375 every 8 connection(s)
I20260618 22:27:39.008013 26244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:39.011853 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26244
I20260618 22:27:39.012252 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:41519
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:42745
--webserver_port=40531
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:27:39.020514 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.009876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:49499 (local address 127.16.243.190:42745)
0618 22:27:39.010013 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.010017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.010979 (+ 962us) server_negotiation.cc:407] Connection header received
0618 22:27:39.012274 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.012278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.012362 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.012440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.014078 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.014935 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.016291 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.016643 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.016753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.020368 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:49499: BlockingRecv error: recv got EOF from 127.16.243.130:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:27:39.021034 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.010287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:49499)
0618 22:27:39.010806 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.010860 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.012066 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.012696 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.012706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.013184 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.013913 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.013930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.015086 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.015089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.016159 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.016169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.016808 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.016812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.016963 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.017623 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.017640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.020158 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":133,"threads_started":1}
W20260618 22:27:39.021631 26376 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:42745 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.243.190:42745: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.027783 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.022115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:44783)
0618 22:27:39.022309 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.022324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.022398 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.022667 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.022670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.022821 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.022922 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.022927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.023973 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.023975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.024539 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.024547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.025105 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.025108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.025185 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.025711 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.025723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.027670 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260618 22:27:39.027957 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.022242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:44783 (local address 127.16.243.190:42745)
0618 22:27:39.022401 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.022405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.022420 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:39.022468 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.022471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.022517 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.022628 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.023030 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.023847 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.024634 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.024980 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.025075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.027837 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:44783: BlockingRecv error: recv got EOF from 127.16.243.130:44783 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.033972 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.028549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:59619)
0618 22:27:39.028727 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.028742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.028817 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.029046 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.029048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.029203 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.029283 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.029288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.030290 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.030293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.030894 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.030903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.031441 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.031443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.031500 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.031960 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.031970 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.033869 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260618 22:27:39.034142 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.028683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59619 (local address 127.16.243.190:42745)
0618 22:27:39.028801 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.028806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.028819 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:39.028908 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.028911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.028955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.029025 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.029380 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.030172 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.030999 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.031349 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.031447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.034030 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:59619: BlockingRecv error: recv got EOF from 127.16.243.130:59619 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:27:39.034289 26376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:27:39.127206 26379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:39.127496 26379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:39.127547 26379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:39.131641 26379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:39.131708 26379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:39.131769 26379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:39.137135 26379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/krb5kdc/oryx_127.16.243.131.keytab
--principal=oryx/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:41519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=40531
--tserver_master_addrs=127.16.243.190:42745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:39.138267 26379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:39.139292 26379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:39.148723 26384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:39.148767 26387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:39.148787 26385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:39.148751 26379 server_base.cc:1034] running on GCE node
I20260618 22:27:39.149322 26379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:39.149960 26379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:39.151124 26379 hybrid_clock.cc:648] HybridClock initialized: now 1781821659151097 us; error 39 us; skew 500 ppm
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821659, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:39.153915 26379 init.cc:377] Logged in from keytab as oryx/127.16.243.131@KRBTEST.COM (short username oryx)
I20260618 22:27:39.154875 26379 webserver.cc:460] Webserver started at http://127.16.243.131:40531/ using document root <none> and password file <none>
I20260618 22:27:39.155409 26379 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:39.155483 26379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:39.159448 26379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:39.161531 26394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:39.162832 26379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:39.162945 26379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "08d3c39c27464fcc9863757a878e1c1b"
format_stamp: "Formatted at 2026-06-18 22:27:38 on dist-test-slave-jm5g"
server_key: "18a9ea1542539fa0025b9a223aabbfca"
server_key_iv: "fa2d6e7dea5014918d398616cb1c5448"
server_key_version: "encryptionkey@0"
I20260618 22:27:39.163487 26379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:39.173413 26379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:39.176522 26379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:39.176689 26379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:39.177259 26379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:39.178081 26379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:39.178125 26379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:39.178169 26379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:39.178195 26379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:39.186744 26379 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:41519
I20260618 22:27:39.186777 26507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:41519 every 8 connection(s)
I20260618 22:27:39.187776 26379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:39.188836 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26379
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.198572 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.189535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.16.243.190:42745)
0618 22:27:39.190012 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.190017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.190351 (+ 334us) server_negotiation.cc:407] Connection header received
0618 22:27:39.190480 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.190482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.190528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.190645 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.191414 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.191952 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.192881 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.193088 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.193172 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.194653 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:39.194659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:39.194673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:39.194701 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:39.196398 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:39.196801 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:39.196806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:39.196811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:39.196941 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:39.197232 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:39.197235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:39.197236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:39.197668 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:39.197872 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:39.198108 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 22:27:39.198354 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
W20260618 22:27:39.199304 26511 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.189898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50119 (local address 127.16.243.190:42745)
0618 22:27:39.190109 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.190119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.191186 (+ 1067us) server_negotiation.cc:407] Connection header received
0618 22:27:39.192116 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.192119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.192192 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.192271 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.193402 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.194075 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.195425 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.195751 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.195846 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.199211 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:50119: BlockingRecv error: recv got EOF from 127.16.243.131:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260618 22:27:39.200065 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.190267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:50119)
0618 22:27:39.191044 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.191095 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.191947 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.192420 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.192426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.192804 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.193278 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.193292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.194215 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.194218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.195294 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.195304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.195913 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.195918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.196085 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.196590 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.196605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.199109 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":185,"threads_started":1}
W20260618 22:27:39.200592 26508 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:42745 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.243.190:42745: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.205811 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.201156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:45413)
0618 22:27:39.201257 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.201272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.201349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.201588 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.201590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.201713 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.201802 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.201806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.202451 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.202453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.203048 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.203056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.203476 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.203478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.203540 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.203982 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.203993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.205646 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260618 22:27:39.205914 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.201221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:45413 (local address 127.16.243.190:42745)
0618 22:27:39.201318 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.201321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.201333 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:27:39.201462 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.201465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.201505 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.201563 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.201879 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.202369 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.203157 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.203384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.203476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.205807 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:45413: BlockingRecv error: recv got EOF from 127.16.243.131:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.211309 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.206311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:53771)
0618 22:27:39.206428 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.206440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.206517 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.206755 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.206757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.206900 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.206969 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.206974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.207662 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.207666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.208175 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.208181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.208582 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.208586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.208657 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.209140 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.209156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.211182 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260618 22:27:39.211441 26246 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.206393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:53771 (local address 127.16.243.190:42745)
0618 22:27:39.206573 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.206577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.206590 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:39.206626 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.206629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.206666 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.206731 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.207070 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.207568 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.208283 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.208492 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.208574 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.211315 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:53771: BlockingRecv error: recv got EOF from 127.16.243.131:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260618 22:27:39.211534 26508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:39 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:39.854992 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:39.849962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:47371)
0618 22:27:39.850245 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:39.850259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:39.850355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:39.850604 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:39.850606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:39.850776 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:39.850896 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.850902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.851606 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.851609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.852186 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:39.852195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.852561 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.852563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:39.852630 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.853075 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:39.853086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:39.854862 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
W20260618 22:27:39.855127 26514 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:39.850055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:47371 (local address 127.16.243.190:42745)
0618 22:27:39.850292 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:39.850296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:39.850309 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:39.850431 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:39.850433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:39.850477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:39.850593 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:39.850988 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.851507 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.852272 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:39.852482 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:39.852548 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:39.855007 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:47371: BlockingRecv error: recv got EOF from 127.16.243.129:47371 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:40.040055 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:40.034933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:43823)
0618 22:27:40.035175 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:40.035187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:40.035295 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:40.035541 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:40.035544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:40.035773 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:40.035867 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.035873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.036596 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.036599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.037165 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.037173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.037589 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.037593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.037673 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.038158 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:40.038171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:40.039920 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
W20260618 22:27:40.040159 26514 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:40.034960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:43823 (local address 127.16.243.190:42745)
0618 22:27:40.035082 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:40.035086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:40.035255 (+ 169us) server_negotiation.cc:407] Connection header received
0618 22:27:40.035392 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:40.035395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:40.035446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:40.035513 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:40.035977 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.036496 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.037268 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.037492 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.037574 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.040052 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:43823: BlockingRecv error: recv got EOF from 127.16.243.130:43823 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:40.218397 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:40.212160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:53037)
0618 22:27:40.212442 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:40.212460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:40.212673 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:40.213022 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:40.213025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:40.213286 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:40.213456 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.213466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.214523 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.214528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.215213 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.215224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.215832 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.215835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.215918 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.216406 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:40.216418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:40.218264 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260618 22:27:40.218513 26514 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:40.212307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:53037 (local address 127.16.243.190:42745)
0618 22:27:40.212447 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:40.212453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:40.212564 (+ 111us) server_negotiation.cc:407] Connection header received
0618 22:27:40.212819 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:40.212822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:40.212884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:40.212964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:40.213604 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.214414 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.215382 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.215712 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.215816 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.218409 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:53037: BlockingRecv error: recv got EOF from 127.16.243.131:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:40 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:40.861025 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:40.855762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:35387)
0618 22:27:40.856016 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:40.856029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:40.856118 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:40.856338 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:40.856340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:40.856522 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:40.856616 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.856621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.857324 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.857326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.857900 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:40.857909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.858307 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.858310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:40.858368 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.858868 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:40.858879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:40.860869 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260618 22:27:40.861156 26518 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:40.855864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:35387 (local address 127.16.243.190:42745)
0618 22:27:40.856091 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:40.856095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:40.856110 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:40.856196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:40.856198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:40.856241 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:40.856320 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:40.856718 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.857247 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.858004 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:40.858225 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:40.858300 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:40.861022 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:35387: BlockingRecv error: recv got EOF from 127.16.243.129:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":58,"threads_started":1}
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:41.046180 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:41.040866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:50463)
0618 22:27:41.041117 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:41.041130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:41.041231 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:41.041471 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:41.041474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:41.041654 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:41.041763 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.041769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.042534 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.042538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.043144 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.043154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.043610 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.043614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.043683 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.044194 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:41.044205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:41.046024 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
W20260618 22:27:41.046295 26518 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:41.040968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:50463 (local address 127.16.243.190:42745)
0618 22:27:41.041128 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:41.041131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:41.041186 (+ 55us) server_negotiation.cc:407] Connection header received
0618 22:27:41.041312 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:41.041315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:41.041360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:41.041428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:41.041863 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.042414 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.043258 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.043475 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.043556 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.046171 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:50463: BlockingRecv error: recv got EOF from 127.16.243.130:50463 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:41.224452 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:41.219151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:41385)
0618 22:27:41.219408 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:41.219423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:41.219514 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:41.219747 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:41.219749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:41.219925 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:41.220021 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.220026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.220901 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.220905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.221612 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.221623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.222032 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.222036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.222112 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.222589 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:41.222601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:41.224330 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260618 22:27:41.224588 26518 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:41.219347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:41385 (local address 127.16.243.190:42745)
0618 22:27:41.219479 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:41.219482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:41.219498 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:27:41.219589 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:41.219592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:41.219639 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:41.219720 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:41.220114 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.220785 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.221707 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.221935 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.222023 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.224476 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:41385: BlockingRecv error: recv got EOF from 127.16.243.131:41385 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:41 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:41.867173 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:41.861816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:40797)
0618 22:27:41.862087 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:41.862102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:41.862199 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:41.862442 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:41.862444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:41.862642 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:41.862739 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.862744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.863528 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.863531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.864124 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:41.864133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.864575 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.864577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:41.864637 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.865137 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:41.865149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:41.866985 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20260618 22:27:41.867447 26522 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:41.861933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:40797 (local address 127.16.243.190:42745)
0618 22:27:41.862163 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:41.862167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:41.862182 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:41.862302 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:41.862304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:41.862348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:41.862422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:41.862862 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.863409 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.864247 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:41.864484 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:41.864563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:41.867133 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:40797: BlockingRecv error: recv got EOF from 127.16.243.129:40797 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:42.052081 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:42.047039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:60037)
0618 22:27:42.047273 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:42.047301 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:42.047397 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:42.047670 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:42.047673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:42.047866 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:42.047970 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.047976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.048708 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.048711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.049268 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.049276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.049695 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.049698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.049774 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.050259 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:42.050273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:42.051927 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
W20260618 22:27:42.052170 26522 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:42.047171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:60037 (local address 127.16.243.190:42745)
0618 22:27:42.047296 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:42.047301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:42.047353 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:27:42.047481 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:42.047484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:42.047543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:42.047629 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:42.048086 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.048608 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.049378 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.049599 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.049683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.052071 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:60037: BlockingRecv error: recv got EOF from 127.16.243.130:60037 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:42.230185 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:42.225092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:47005)
0618 22:27:42.225353 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:42.225367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:42.225476 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:42.225707 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:42.225710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:42.225903 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:42.226001 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.226007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.226788 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.226791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.227368 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.227377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.227789 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.227792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.227865 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.228351 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:42.228363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:42.230058 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
W20260618 22:27:42.230312 26522 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:42.225175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:47005 (local address 127.16.243.190:42745)
0618 22:27:42.225305 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:42.225310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:42.225439 (+ 129us) server_negotiation.cc:407] Connection header received
0618 22:27:42.225563 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:42.225566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:42.225614 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:42.225676 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:42.226116 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.226682 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.227475 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.227693 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.227779 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.230205 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:47005: BlockingRecv error: recv got EOF from 127.16.243.131:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:42 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:42.877825 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:42.868104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:39647)
0618 22:27:42.868375 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:42.868397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:42.868505 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:42.868751 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:42.868753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:42.868936 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:42.869036 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.869042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.869970 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.869975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.870682 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:42.870694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.871157 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.871160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:42.871256 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.875774 (+ 4518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:42.875788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:42.877700 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260618 22:27:42.878026 26526 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:42.868181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:39647 (local address 127.16.243.190:42745)
0618 22:27:42.868462 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:42.868466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:42.868483 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:27:42.868590 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:42.868593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:42.868637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:42.868731 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:42.869177 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.869807 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.870786 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:42.871025 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:42.871109 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:42.877892 (+ 6783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:39647: BlockingRecv error: recv got EOF from 127.16.243.129:39647 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:43.057794 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:43.052852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:47891)
0618 22:27:43.053088 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:43.053100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:43.053200 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:43.053422 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:43.053424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:43.053592 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:43.053691 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.053696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.054394 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.054396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.054991 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.055001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.055442 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.055444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.055502 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.055965 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:43.055976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:43.057675 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
W20260618 22:27:43.057984 26526 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:43.052913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:47891 (local address 127.16.243.190:42745)
0618 22:27:43.053088 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:43.053093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:43.053158 (+ 65us) server_negotiation.cc:407] Connection header received
0618 22:27:43.053281 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:43.053283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:43.053332 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:43.053407 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:43.053779 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.054312 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.055096 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.055342 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.055437 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.057852 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:47891: BlockingRecv error: recv got EOF from 127.16.243.130:47891 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:43.236150 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:43.230981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:44035)
0618 22:27:43.231231 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:43.231271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:43.231366 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:43.231574 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:43.231576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:43.231746 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:43.231840 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.231844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.232519 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.232521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.233065 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.233072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.233463 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.233467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.233539 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.234061 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:43.234074 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:43.236033 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
W20260618 22:27:43.236331 26526 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:43.231051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:44035 (local address 127.16.243.190:42745)
0618 22:27:43.231163 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:43.231167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:43.231319 (+ 152us) server_negotiation.cc:407] Connection header received
0618 22:27:43.231437 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:43.231440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:43.231484 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:43.231553 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:43.231923 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.232441 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.233142 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.233346 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.233413 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.236201 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:44035: BlockingRecv error: recv got EOF from 127.16.243.131:44035 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:43 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:43.884092 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:43.878793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:38261)
0618 22:27:43.879174 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:43.879190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:43.879293 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:43.879567 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:43.879569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:43.879760 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:43.879877 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.879887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.880614 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.880617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.881185 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:43.881193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.881642 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.881644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:43.881701 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.882196 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:43.882208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:43.883974 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
W20260618 22:27:43.884222 26530 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:43.878858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:38261 (local address 127.16.243.190:42745)
0618 22:27:43.879181 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:43.879185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:43.879248 (+ 63us) server_negotiation.cc:407] Connection header received
0618 22:27:43.879382 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:43.879385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:43.879438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:43.879522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:43.879986 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.880523 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.881314 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:43.881552 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:43.881638 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:43.884103 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:38261: BlockingRecv error: recv got EOF from 127.16.243.129:38261 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:44.063777 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:44.058540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:47327)
0618 22:27:44.058819 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:44.058832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:44.058925 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:44.059136 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:44.059138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:44.059314 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:44.059407 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.059412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.060117 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.060119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.060688 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.060696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.061164 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.061166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.061224 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.061704 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:44.061715 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:44.063649 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
W20260618 22:27:44.063937 26530 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:44.058715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:47327 (local address 127.16.243.190:42745)
0618 22:27:44.058847 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:44.058851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:44.058894 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:27:44.058999 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:44.059001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:44.059046 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:44.059111 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:44.059497 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.060033 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.060819 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.061061 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.061144 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.063804 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:47327: BlockingRecv error: recv got EOF from 127.16.243.130:47327 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:44.242035 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:44.236963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:42235)
0618 22:27:44.237202 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:44.237217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:44.237313 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:44.237547 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:44.237549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:44.237721 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:44.237813 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.237818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.238539 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.238541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.239137 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.239144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.239545 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.239547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.239615 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.240067 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:44.240078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:44.241911 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
W20260618 22:27:44.242183 26530 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:44.237042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:42235 (local address 127.16.243.190:42745)
0618 22:27:44.237183 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:44.237187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:44.237279 (+ 92us) server_negotiation.cc:407] Connection header received
0618 22:27:44.237385 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:44.237404 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:44.237450 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:44.237524 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:44.237899 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.238450 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.239224 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.239453 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.239529 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.242064 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:42235: BlockingRecv error: recv got EOF from 127.16.243.131:42235 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:44 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:44.890458 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:44.884999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:37609)
0618 22:27:44.885272 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:44.885288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:44.885402 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:44.885780 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:44.885783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:44.886019 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:44.886122 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.886130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.887062 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.887066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.887635 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:44.887643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.888069 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.888071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:44.888130 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.888625 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:44.888635 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:44.890321 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
W20260618 22:27:44.890664 26534 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:44.885135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37609 (local address 127.16.243.190:42745)
0618 22:27:44.885500 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:44.885505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:44.885520 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:44.885565 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:44.885568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:44.885627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:44.885730 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:44.886257 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.886956 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.887748 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:44.887980 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:44.888062 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:44.890494 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:37609: BlockingRecv error: recv got EOF from 127.16.243.129:37609 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:45.069798 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:45.064590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:47769)
0618 22:27:45.064841 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:45.064855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:45.064956 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:45.065173 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:45.065175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:45.065351 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:45.065459 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.065465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.066206 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.066209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.066831 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.066840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.067288 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.067292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.067357 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.067846 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:45.067857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:45.069646 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
W20260618 22:27:45.069880 26534 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:45.064679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:47769 (local address 127.16.243.190:42745)
0618 22:27:45.064826 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:45.064829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:45.064920 (+ 91us) server_negotiation.cc:407] Connection header received
0618 22:27:45.065028 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:45.065030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:45.065075 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:45.065149 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:45.065566 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.066094 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.066923 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.067165 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.067245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.069779 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:47769: BlockingRecv error: recv got EOF from 127.16.243.130:47769 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:45.248067 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:45.242843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:55597)
0618 22:27:45.243099 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:45.243114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:45.243225 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:45.243484 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:45.243487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:45.243721 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:45.243812 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.243817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.244582 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.244585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.245142 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.245149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.245577 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.245580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.245657 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.246110 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:45.246122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:45.247918 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260618 22:27:45.248172 26534 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:45.242904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:55597 (local address 127.16.243.190:42745)
0618 22:27:45.243052 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:45.243057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:45.243183 (+ 126us) server_negotiation.cc:407] Connection header received
0618 22:27:45.243318 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:45.243321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:45.243372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:45.243437 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:45.243927 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.244475 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.245251 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.245478 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.245562 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.248071 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:55597: BlockingRecv error: recv got EOF from 127.16.243.131:55597 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:45 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:45.897848 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:45.891336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:37059)
0618 22:27:45.891650 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:45.891665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:45.891796 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:45.892086 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:45.892089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:45.892295 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:45.892387 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.892393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.893209 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.893213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.894151 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:45.894164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.894747 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.894751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:45.894848 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.895392 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:45.895406 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:45.897671 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
W20260618 22:27:45.898173 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:45.891386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37059 (local address 127.16.243.190:42745)
0618 22:27:45.891630 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:45.891634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:45.891739 (+ 105us) server_negotiation.cc:407] Connection header received
0618 22:27:45.891889 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:45.891892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:45.891960 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:45.892043 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:45.892504 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.893083 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.894264 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:45.894643 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:45.894721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:45.897893 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:37059: BlockingRecv error: recv got EOF from 127.16.243.129:37059 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:46.076099 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:46.070779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:39835)
0618 22:27:46.071074 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:46.071087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:46.071199 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:46.071435 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:46.071437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:46.071662 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:46.071754 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.071760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.072546 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.072550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.073120 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.073128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.073578 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.073582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.073659 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.074169 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:46.074183 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:46.075908 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
W20260618 22:27:46.076167 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:46.070841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39835 (local address 127.16.243.190:42745)
0618 22:27:46.070985 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:46.070990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:46.071174 (+ 184us) server_negotiation.cc:407] Connection header received
0618 22:27:46.071288 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:46.071291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:46.071340 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:46.071401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:46.071869 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.072443 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.073234 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.073474 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.073561 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.076065 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:39835: BlockingRecv error: recv got EOF from 127.16.243.130:39835 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:46.254112 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:46.248810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:57117)
0618 22:27:46.249138 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:46.249151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:46.249264 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:46.249529 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:46.249532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:46.249732 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:46.249852 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.249858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.250614 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.250618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.251186 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.251194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.251650 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.251653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.251740 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.252242 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:46.252254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:46.253949 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
W20260618 22:27:46.254206 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:46.248908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:57117 (local address 127.16.243.190:42745)
0618 22:27:46.249044 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:46.249049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:46.249220 (+ 171us) server_negotiation.cc:407] Connection header received
0618 22:27:46.249381 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:46.249384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:46.249432 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:46.249497 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:46.249966 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.250498 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.251294 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.251528 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.251613 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.254102 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:57117: BlockingRecv error: recv got EOF from 127.16.243.131:57117 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:46 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:46.903915 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:46.898632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:52505)
0618 22:27:46.898894 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:46.898908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:46.899011 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:46.899273 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:46.899277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:46.899469 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:46.899565 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.899571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.900310 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.900314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.900878 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:46.900886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.901304 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.901308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:46.901386 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.901884 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:46.901896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:46.903755 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
W20260618 22:27:46.904036 26542 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:46.898783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:52505 (local address 127.16.243.190:42745)
0618 22:27:46.899010 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:46.899015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:46.899029 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:46.899102 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:46.899104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:46.899150 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:46.899228 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:46.899678 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.900207 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.900987 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:46.901210 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:46.901292 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:46.903908 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:52505: BlockingRecv error: recv got EOF from 127.16.243.129:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:47.082216 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:47.076879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:57827)
0618 22:27:47.077120 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:47.077133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:47.077230 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:47.077468 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:47.077471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:47.077667 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:47.077779 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.077786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.078691 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.078694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.079317 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.079328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.079757 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.079759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.079816 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.080335 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:47.080346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:47.082094 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
W20260618 22:27:47.082387 26542 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:47.076983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57827 (local address 127.16.243.190:42745)
0618 22:27:47.077106 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:47.077110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:47.077229 (+ 119us) server_negotiation.cc:407] Connection header received
0618 22:27:47.077307 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:47.077310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:47.077382 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:47.077445 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:47.077884 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.078598 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.079414 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.079672 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.079747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.082254 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:57827: BlockingRecv error: recv got EOF from 127.16.243.130:57827 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:47.260159 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:47.254908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:50383)
0618 22:27:47.255262 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:47.255275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:47.255392 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:47.255648 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:47.255651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:47.255844 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:47.255949 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.255955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.256671 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.256676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.257233 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.257241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.257683 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.257686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.257766 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.258224 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:47.258235 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:47.260033 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
W20260618 22:27:47.260289 26542 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:47.255013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50383 (local address 127.16.243.190:42745)
0618 22:27:47.255181 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:47.255185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:47.255356 (+ 171us) server_negotiation.cc:407] Connection header received
0618 22:27:47.255502 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:47.255505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:47.255554 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:47.255619 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:47.256057 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.256571 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.257341 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.257582 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.257666 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.260181 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:50383: BlockingRecv error: recv got EOF from 127.16.243.131:50383 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:47 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:47.912896 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:47.904671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:48421)
0618 22:27:47.904929 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:47.904942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:47.905037 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:47.905254 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:47.905256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:47.905453 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:47.905551 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.905557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.906280 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.906283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.906915 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:47.906925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.907333 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.907335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:47.907394 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.910873 (+ 3479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:47.910887 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:47.912762 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
W20260618 22:27:47.913060 26546 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:47.904738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:48421 (local address 127.16.243.190:42745)
0618 22:27:47.904958 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:47.904962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:47.904992 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:27:47.905111 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:47.905114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:47.905156 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:47.905230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:47.905642 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.906181 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.907011 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:47.907228 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:47.907302 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:47.912929 (+ 5627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:48421: BlockingRecv error: recv got EOF from 127.16.243.129:48421 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:48.088289 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:48.082940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:43165)
0618 22:27:48.083199 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:48.083218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:48.083328 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:48.083543 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:48.083545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:48.083728 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:48.083821 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.083827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.084614 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.084616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.085176 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.085183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.085634 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.085636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.085698 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.086186 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:48.086197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:48.088149 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20260618 22:27:48.088405 26546 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:48.083066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:43165 (local address 127.16.243.190:42745)
0618 22:27:48.083218 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:48.083221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:48.083283 (+ 62us) server_negotiation.cc:407] Connection header received
0618 22:27:48.083398 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:48.083400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:48.083452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:48.083518 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:48.083951 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.084514 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.085292 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.085531 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.085619 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.088302 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:43165: BlockingRecv error: recv got EOF from 127.16.243.130:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:48.266638 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:48.260882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:58387)
0618 22:27:48.261206 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:48.261223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:48.261352 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:48.261627 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:48.261631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:48.261849 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:48.261949 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.261956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.262787 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.262791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.263395 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.263405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.263891 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.263896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.263985 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.264484 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:48.264497 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:48.266429 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
W20260618 22:27:48.266712 26546 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:48.260993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:58387 (local address 127.16.243.190:42745)
0618 22:27:48.261163 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:48.261168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:48.261305 (+ 137us) server_negotiation.cc:407] Connection header received
0618 22:27:48.261465 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:48.261469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:48.261520 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:48.261590 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:48.262072 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.262681 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.263516 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.263757 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.263855 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.266609 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:58387: BlockingRecv error: recv got EOF from 127.16.243.131:58387 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:48 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:48.919278 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:48.913827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:53431)
0618 22:27:48.914119 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:48.914133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:48.914226 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:48.914490 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:48.914493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:48.914700 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:48.914801 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.914807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.915556 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.915559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.916151 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:48.916161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.916626 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.916630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:48.916707 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.917239 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:48.917251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:48.919127 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
W20260618 22:27:48.919390 26550 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:48.913992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:53431 (local address 127.16.243.190:42745)
0618 22:27:48.914224 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:48.914228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:48.914243 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:27:48.914289 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:48.914291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:48.914333 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:48.914408 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:48.914902 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.915436 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.916274 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:48.916510 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:48.916590 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:48.919275 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:53431: BlockingRecv error: recv got EOF from 127.16.243.129:53431 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:49.094664 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:49.089179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:55743)
0618 22:27:49.089470 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:49.089486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:49.089608 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:49.089857 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:49.089860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:49.090056 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:49.090147 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.090153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.090948 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.090952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.091532 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.091541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.091960 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.091964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.092075 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.092607 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:49.092619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:49.094484 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
W20260618 22:27:49.094765 26550 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:49.089294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55743 (local address 127.16.243.190:42745)
0618 22:27:49.089473 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:49.089477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:49.089565 (+ 88us) server_negotiation.cc:407] Connection header received
0618 22:27:49.089698 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:49.089701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:49.089749 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:49.089820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:49.090263 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.090843 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.091642 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.091861 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.091945 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.094660 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:55743: BlockingRecv error: recv got EOF from 127.16.243.130:55743 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:49.272833 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:49.267590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:59707)
0618 22:27:49.267850 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:49.267864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:49.267968 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:49.268184 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:49.268186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:49.268356 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:49.268448 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.268454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.269210 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.269212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.269835 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.269845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.270329 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.270332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.270395 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.270907 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:49.270922 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:49.272704 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
W20260618 22:27:49.272980 26550 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:49.267724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:59707 (local address 127.16.243.190:42745)
0618 22:27:49.267867 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:49.267870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:49.267915 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:27:49.268043 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:49.268045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:49.268091 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:49.268162 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:49.268568 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.269120 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.269957 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.270209 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.270306 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.272846 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:59707: BlockingRecv error: recv got EOF from 127.16.243.131:59707 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:49 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:49.924966 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:49.920073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:56399)
0618 22:27:49.920331 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:49.920345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:49.920440 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:49.920674 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:49.920676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:49.920851 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:49.920949 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.920968 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.921663 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.921666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.922226 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:49.922233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.922634 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.922637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:49.922698 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.923174 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:49.923185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:49.924821 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260618 22:27:49.925098 26554 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:49.920141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:56399 (local address 127.16.243.190:42745)
0618 22:27:49.920416 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:49.920419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:49.920433 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:49.920532 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:49.920534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:49.920577 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:49.920645 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:49.921052 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.921584 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.922311 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:49.922532 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:49.922621 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:49.924968 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:56399: BlockingRecv error: recv got EOF from 127.16.243.129:56399 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:50.100607 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:50.095459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:33833)
0618 22:27:50.095724 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:50.095738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:50.095848 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:50.096086 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:50.096089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:50.096283 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:50.096372 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.096377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.097128 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.097132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.097688 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.097696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.098130 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.098133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.098211 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.098753 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:50.098764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:50.100472 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":70,"threads_started":1}
W20260618 22:27:50.100719 26554 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:50.095535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:33833 (local address 127.16.243.190:42745)
0618 22:27:50.095666 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:50.095670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:50.095801 (+ 131us) server_negotiation.cc:407] Connection header received
0618 22:27:50.095933 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:50.095936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:50.095985 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:50.096056 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:50.096480 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.097023 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.097797 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.098027 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.098114 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.100617 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:33833: BlockingRecv error: recv got EOF from 127.16.243.130:33833 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:50.278362 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:50.273541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:42565)
0618 22:27:50.273779 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:50.273792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:50.273886 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:50.274098 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:50.274100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:50.274267 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:50.274357 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.274361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.275061 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.275063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.275609 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.275616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.275981 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.275984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.276053 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.276514 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:50.276524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:50.278229 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
W20260618 22:27:50.278486 26554 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:50.273628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:42565 (local address 127.16.243.190:42745)
0618 22:27:50.273746 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:50.273749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:50.273852 (+ 103us) server_negotiation.cc:407] Connection header received
0618 22:27:50.273956 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:50.273958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:50.274005 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:50.274075 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:50.274443 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.274989 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.275688 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.275903 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.275970 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.278378 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:42565: BlockingRecv error: recv got EOF from 127.16.243.131:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:50 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:50.930751 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:50.925727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:55373)
0618 22:27:50.926010 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:50.926026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:50.926140 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:50.926431 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:50.926435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:50.926667 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:50.926767 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.926774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.927455 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.927457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.928009 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:50.928016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.928370 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.928372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:50.928427 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.928922 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:50.928935 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:50.930638 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
W20260618 22:27:50.930888 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:50.925790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:55373 (local address 127.16.243.190:42745)
0618 22:27:50.926049 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:50.926053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:50.926081 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:27:50.926237 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:50.926240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:50.926296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:50.926390 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:50.926858 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.927381 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.928087 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:50.928295 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:50.928361 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:50.930768 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:55373: BlockingRecv error: recv got EOF from 127.16.243.129:55373 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:51.106276 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:51.101317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:57875)
0618 22:27:51.101604 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:51.101616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:51.101709 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:51.101911 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:51.101913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:51.102085 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:51.102177 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.102182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.102889 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.102892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.103444 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.103451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.103866 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.103868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.103925 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.104411 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:51.104422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:51.106157 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
W20260618 22:27:51.106416 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:51.101402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57875 (local address 127.16.243.190:42745)
0618 22:27:51.101538 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:51.101542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:51.101675 (+ 133us) server_negotiation.cc:407] Connection header received
0618 22:27:51.101778 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:51.101780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:51.101823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:51.101888 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:51.102255 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.102806 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.103554 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.103776 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.103856 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.106294 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:57875: BlockingRecv error: recv got EOF from 127.16.243.130:57875 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:51.284300 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:51.279141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:41321)
0618 22:27:51.279398 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:51.279411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:51.279517 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:51.279716 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:51.279718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:51.279891 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:51.279986 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.279991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.280692 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.280695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.281256 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.281264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.281674 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.281677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.281751 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.282336 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:51.282348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:51.284171 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
W20260618 22:27:51.284411 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:51.279197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:41321 (local address 127.16.243.190:42745)
0618 22:27:51.279321 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:51.279324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:51.279477 (+ 153us) server_negotiation.cc:407] Connection header received
0618 22:27:51.279580 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:51.279582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:51.279627 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:51.279691 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:51.280070 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.280580 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.281352 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.281556 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.281622 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.284296 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:41321: BlockingRecv error: recv got EOF from 127.16.243.131:41321 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:51 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:51.937362 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:51.931690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:38197)
0618 22:27:51.931990 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:51.932004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:51.932100 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:51.932337 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:51.932339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:51.932521 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:51.932620 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.932625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.933379 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.933383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.934183 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:51.934196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.934783 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.934786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:51.934868 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.935362 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:51.935375 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:51.937243 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
W20260618 22:27:51.937546 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:51.931831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:38197 (local address 127.16.243.190:42745)
0618 22:27:51.932060 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:51.932064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:51.932080 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:27:51.932184 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:51.932187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:51.932233 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:51.932309 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:51.932715 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.933245 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.934316 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:51.934681 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:51.934762 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:51.937389 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:38197: BlockingRecv error: recv got EOF from 127.16.243.129:38197 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:52.112954 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:52.107130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:59333)
0618 22:27:52.107431 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:52.107445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:52.107547 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:52.107879 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:52.107883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:52.108109 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:52.108209 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.108215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.109091 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.109094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.109706 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.109716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.110235 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.110238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.110311 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.110864 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:52.110875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:52.112823 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
W20260618 22:27:52.113107 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:52.107347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59333 (local address 127.16.243.190:42745)
0618 22:27:52.107502 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:52.107506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:52.107522 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:27:52.107644 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:52.107647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:52.107701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:52.107800 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:52.108345 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.108982 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.109851 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.110089 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.110179 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.112981 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:59333: BlockingRecv error: recv got EOF from 127.16.243.130:59333 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:52.290726 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:52.285127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:53131)
0618 22:27:52.285393 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:52.285408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:52.285511 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:52.285792 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:52.285795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:52.285999 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:52.286097 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.286102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.287019 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.287023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.287631 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.287640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.288062 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.288065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.288131 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.288616 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:52.288630 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:52.290545 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
W20260618 22:27:52.290838 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:52.285244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:53131 (local address 127.16.243.190:42745)
0618 22:27:52.285397 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:52.285401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:52.285461 (+ 60us) server_negotiation.cc:407] Connection header received
0618 22:27:52.285588 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:52.285591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:52.285658 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:52.285734 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:52.286205 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.286913 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.287731 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.287945 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.288023 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.290719 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:53131: BlockingRecv error: recv got EOF from 127.16.243.131:53131 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:52 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:52.946041 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:52.938234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:59137)
0618 22:27:52.938492 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:52.938509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:52.938647 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:52.938910 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:52.938912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:52.939100 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:52.939193 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.939198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.939910 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.939913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.940485 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:52.940492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.940912 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.940915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:52.940970 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.944050 (+ 3080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:52.944075 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:52.945914 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
W20260618 22:27:52.946190 26566 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:52.938351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:59137 (local address 127.16.243.190:42745)
0618 22:27:52.938698 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:52.938701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:52.938715 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:52.938756 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:52.938759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:52.938804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:52.938884 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:52.939295 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.939825 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.940587 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:52.940824 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:52.940899 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:52.946064 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:59137: BlockingRecv error: recv got EOF from 127.16.243.129:59137 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":75,"threads_started":1}
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:53.119626 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:53.113835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:60483)
0618 22:27:53.114127 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:53.114140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:53.114244 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:53.114570 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:53.114573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:53.114761 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:53.114863 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.114869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.115658 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.115661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.116268 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.116277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.116790 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.116794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.116869 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.117425 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:53.117437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:53.119492 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
W20260618 22:27:53.119801 26566 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:53.113969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:60483 (local address 127.16.243.190:42745)
0618 22:27:53.114116 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:53.114120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:53.114207 (+ 87us) server_negotiation.cc:407] Connection header received
0618 22:27:53.114374 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:53.114377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:53.114430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:53.114519 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:53.114990 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.115529 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.116418 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.116654 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.116743 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.119660 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:60483: BlockingRecv error: recv got EOF from 127.16.243.130:60483 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.131@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:53.297709 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:53.291658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.131:35735)
0618 22:27:53.291937 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:53.291960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:53.292070 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:53.292364 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:53.292366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:53.292571 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:53.292671 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.292676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.293477 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.293481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.294045 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.294054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.294536 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.294539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.294664 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.295224 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:53.295238 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:53.297546 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
W20260618 22:27:53.297818 26566 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:53.291743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:35735 (local address 127.16.243.190:42745)
0618 22:27:53.291943 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:53.291948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:53.292031 (+ 83us) server_negotiation.cc:407] Connection header received
0618 22:27:53.292147 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:53.292150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:53.292217 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:53.292294 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:53.292787 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.293355 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.294161 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.294387 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.294493 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.297717 (+ 3224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.131:35735: BlockingRecv error: recv got EOF from 127.16.243.131:35735 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:53 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.129@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:53.952853 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:53.947026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.129:48869)
0618 22:27:53.947300 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:53.947318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:53.947411 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:53.947624 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:53.947625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:53.947801 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:53.947898 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.947903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.948774 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.948777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.949418 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:53.949430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.949939 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.949943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:53.950016 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.950591 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:53.950609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:53.952705 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
W20260618 22:27:53.953003 26570 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:53.947159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:48869 (local address 127.16.243.190:42745)
0618 22:27:53.947377 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:53.947381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:53.947394 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:53.947480 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:53.947482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:53.947523 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:53.947598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:53.948050 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.948593 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.949558 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:53.949806 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:53.949900 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:53.952867 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.129:48869: BlockingRecv error: recv got EOF from 127.16.243.129:48869 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":57,"threads_started":1}
Jun 18 22:27:54 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:27:54 dist-test-slave-jm5g krb5kdc[25511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.243.130@KRBTEST.COM for oryx/127.16.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:27:54.125943 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.120541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42745 (local address 127.16.243.130:38319)
0618 22:27:54.120794 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:54.120807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:54.120899 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:54.121167 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:54.121170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:54.121356 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:54.121475 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.121481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.122235 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.122238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.122910 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.122919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.123346 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.123349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.123444 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.123921 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:54.123932 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:54.125777 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.243.190:42745: Server oryx/127.16.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
W20260618 22:27:54.126075 26570 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:54.120652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:38319 (local address 127.16.243.190:42745)
0618 22:27:54.120796 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:54.120800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:54.120869 (+ 69us) server_negotiation.cc:407] Connection header received
0618 22:27:54.120973 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:54.120975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:54.121017 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:54.121104 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:54.121587 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.122118 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.123010 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.123252 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.123330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.125971 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.243.130:38319: BlockingRecv error: recv got EOF from 127.16.243.130:38319 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:27:54.198807 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26094
I20260618 22:27:54.203408 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26244
I20260618 22:27:54.207883 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26379
I20260618 22:27:54.212054 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26024
2026-06-18T22:27:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17873 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-18T22:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:54Z Disabled control of system clock
I20260618 22:27:54.239651 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35299
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:27:54.360211 26575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:54.360494 26575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:54.360529 26575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:54.364778 26575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:54.364861 26575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:54.364876 26575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:54.364892 26575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:54.364912 26575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:54.369879 26575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:35299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:54.371021 26575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:54.372570 26575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:54.381871 26580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.381932 26575 server_base.cc:1034] running on GCE node
W20260618 22:27:54.381987 26583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.382175 26581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.382856 26575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:54.384097 26575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:54.385272 26575 hybrid_clock.cc:648] HybridClock initialized: now 1781821674385246 us; error 41 us; skew 500 ppm
I20260618 22:27:54.387207 26575 webserver.cc:460] Webserver started at http://127.16.243.190:39263/ using document root <none> and password file <none>
I20260618 22:27:54.387804 26575 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:54.387872 26575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:54.388082 26575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:54.390203 26575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "4c0b907746604c39b75f89205a406052"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "9f60104f376f772b478b8aa0fdafd849"
server_key_iv: "c7b6074f26a224454116f6246d4a03ab"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.390785 26575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "4c0b907746604c39b75f89205a406052"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "9f60104f376f772b478b8aa0fdafd849"
server_key_iv: "c7b6074f26a224454116f6246d4a03ab"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.394610 26575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:27:54.397167 26589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.398639 26575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:54.398772 26575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "4c0b907746604c39b75f89205a406052"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "9f60104f376f772b478b8aa0fdafd849"
server_key_iv: "c7b6074f26a224454116f6246d4a03ab"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.398885 26575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:54.411417 26575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:54.412144 26575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:54.412328 26575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:54.418995 26575 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:35299
I20260618 22:27:54.419005 26641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:35299 every 8 connection(s)
I20260618 22:27:54.419975 26575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:54.423537 26642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:54.425443 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26575
I20260618 22:27:54.425558 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:54.425859 17358 external_mini_cluster.cc:1428] Setting key b54a3a651d455d016da1a08ad785f263
I20260618 22:27:54.430757 26642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052: Bootstrap starting.
I20260618 22:27:54.433241 26642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:54.434145 26642 log.cc:824] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:54.435842 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.427470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.16.243.190:35299)
0618 22:27:54.428210 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:54.428227 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:54.428272 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:27:54.428977 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:54.428997 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:54.429450 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:54.429798 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:54.430252 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.431375 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.432334 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.432836 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.432988 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.433856 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:54.433861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:54.433875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:54.434333 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:54.434400 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:54.434422 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:54.434506 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:54.434737 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 22:27:54.435091 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":147,"threads_started":1}
I20260618 22:27:54.436319 26642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052: No bootstrap required, opened a new log
I20260618 22:27:54.440222 26642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } }
I20260618 22:27:54.440470 26642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:54.440526 26642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0b907746604c39b75f89205a406052, State: Initialized, Role: FOLLOWER
I20260618 22:27:54.441006 26642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } }
I20260618 22:27:54.441166 26642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:54.441258 26642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:54.441372 26642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:54.442349 26642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } }
I20260618 22:27:54.442749 26642 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0b907746604c39b75f89205a406052; no voters:
I20260618 22:27:54.443061 26642 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:54.443163 26647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:54.443444 26647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [term 1 LEADER]: Becoming Leader. State: Replica: 4c0b907746604c39b75f89205a406052, State: Running, Role: LEADER
I20260618 22:27:54.443775 26647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } }
I20260618 22:27:54.444336 26642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:54.445386 26649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0b907746604c39b75f89205a406052. Latest consensus state: current_term: 1 leader_uuid: "4c0b907746604c39b75f89205a406052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } } }
I20260618 22:27:54.445515 26649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:54.445845 26648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0b907746604c39b75f89205a406052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0b907746604c39b75f89205a406052" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35299 } } }
I20260618 22:27:54.445901 26656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:54.446012 26648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:54.449221 26656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:54.455075 26656 catalog_manager.cc:1347] Generated new cluster ID: 47a496b9fa3b4e11997a9d796bbe30d7
I20260618 22:27:54.455150 26656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:54.463423 26656 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:54.464596 26656 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:54.470342 26656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0b907746604c39b75f89205a406052: Generated new TSK 0
I20260618 22:27:54.471277 26656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:54.475188 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--builtin_ntp_servers=127.16.243.148:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260618 22:27:54.590034 26666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:54.590307 26666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:54.590361 26666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:54.594532 26666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:54.594650 26666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:54.594731 26666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:54.599833 26666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:54.601009 26666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:54.602396 26666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:54.611853 26674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.611902 26672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.611853 26671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.612130 26666 server_base.cc:1034] running on GCE node
I20260618 22:27:54.612550 26666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:54.613193 26666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:54.614355 26666 hybrid_clock.cc:648] HybridClock initialized: now 1781821674614335 us; error 39 us; skew 500 ppm
I20260618 22:27:54.616225 26666 webserver.cc:460] Webserver started at http://127.16.243.129:34949/ using document root <none> and password file <none>
I20260618 22:27:54.616787 26666 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:54.616856 26666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:54.617055 26666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:54.619189 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "14fb66f5c77d4027afa386660c4d8240"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "2637b51871aeedc70015e5c2bc3b8a30"
server_key_iv: "b18a13143191da68bb66c2b62763cf85"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.619731 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "14fb66f5c77d4027afa386660c4d8240"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "2637b51871aeedc70015e5c2bc3b8a30"
server_key_iv: "b18a13143191da68bb66c2b62763cf85"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.623462 26666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:27:54.626047 26680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.627377 26666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:54.627492 26666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "14fb66f5c77d4027afa386660c4d8240"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "2637b51871aeedc70015e5c2bc3b8a30"
server_key_iv: "b18a13143191da68bb66c2b62763cf85"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.627607 26666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:54.651116 26666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:54.651854 26666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:54.652040 26666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:54.652643 26666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:54.653635 26666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:54.653684 26666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.653738 26666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:54.653767 26666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.662690 26666 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:40561
I20260618 22:27:54.662714 26793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:40561 every 8 connection(s)
I20260618 22:27:54.663707 26666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:54.672032 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26666
I20260618 22:27:54.672142 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:54.672447 17358 external_mini_cluster.cc:1428] Setting key 0c1d9f325b84c7ed2a3fcfe89611a01a
I20260618 22:27:54.673197 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.665658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:56133 (local address 127.16.243.190:35299)
0618 22:27:54.665780 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:54.665783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:54.666968 (+ 1185us) server_negotiation.cc:407] Connection header received
0618 22:27:54.668389 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:54.668393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:54.668453 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:54.668524 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:54.669480 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.670044 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.671241 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.671460 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.671560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.672708 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:54.672712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:54.672715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:54.672731 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:54.672754 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:54.672765 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:54.672837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:54.672993 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:27:54.673059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:54.674220 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.665994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35299 (local address 127.16.243.129:56133)
0618 22:27:54.666805 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:54.666852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:54.668161 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:54.668738 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:54.668745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:54.668837 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:54.669343 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.669355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.670173 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.670176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.671128 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.671139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.671609 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.671613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.671769 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.672414 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:54.672415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:54.672457 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:54.672525 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:54.672845 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:54.672852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:54.672857 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:54.672981 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 22:27:54.673263 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260618 22:27:54.674413 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--builtin_ntp_servers=127.16.243.148:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:27:54.675379 26794 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35299
I20260618 22:27:54.675645 26794 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:54.676185 26794 heartbeater.cc:511] Master 127.16.243.190:35299 requested a full tablet report, sending...
I20260618 22:27:54.677793 26604 ts_manager.cc:194] Registered new tserver with Master: 14fb66f5c77d4027afa386660c4d8240 (127.16.243.129:40561)
I20260618 22:27:54.679469 26604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.129:56133
W20260618 22:27:54.785218 26797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:54.785513 26797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:54.785550 26797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:54.789700 26797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:54.789769 26797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:54.789825 26797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:54.794718 26797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:54.795800 26797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:54.797122 26797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:54.806520 26803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.806623 26805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.806532 26802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.806697 26797 server_base.cc:1034] running on GCE node
I20260618 22:27:54.807225 26797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:54.807873 26797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:54.809042 26797 hybrid_clock.cc:648] HybridClock initialized: now 1781821674809024 us; error 36 us; skew 500 ppm
I20260618 22:27:54.811064 26797 webserver.cc:460] Webserver started at http://127.16.243.130:39091/ using document root <none> and password file <none>
I20260618 22:27:54.811673 26797 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:54.811748 26797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:54.811970 26797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:54.814162 26797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "0f64a65258974fb79f92599436bbd3c2"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "66ef390f11ab0163da67e602524bbf71"
server_key_iv: "831ee05bbb8f7ca9f6ba364683390140"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.814802 26797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "0f64a65258974fb79f92599436bbd3c2"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "66ef390f11ab0163da67e602524bbf71"
server_key_iv: "831ee05bbb8f7ca9f6ba364683390140"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.818786 26797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:27:54.821537 26811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.822924 26797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:54.823045 26797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "0f64a65258974fb79f92599436bbd3c2"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "66ef390f11ab0163da67e602524bbf71"
server_key_iv: "831ee05bbb8f7ca9f6ba364683390140"
server_key_version: "encryptionkey@0"
I20260618 22:27:54.823158 26797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:54.843741 26797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:54.844537 26797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:54.844709 26797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:54.845330 26797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:54.846282 26797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:54.846338 26797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.846397 26797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:54.846423 26797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:54.854676 26797 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:34923
I20260618 22:27:54.854715 26924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:34923 every 8 connection(s)
I20260618 22:27:54.855607 26797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:54.861231 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26797
I20260618 22:27:54.861333 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:54.861588 17358 external_mini_cluster.cc:1428] Setting key 4cc513253b812b49f04dcc287861955b
I20260618 22:27:54.863596 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--builtin_ntp_servers=127.16.243.148:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:27:54.865356 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.857447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59953 (local address 127.16.243.190:35299)
0618 22:27:54.857603 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:54.857607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:54.858721 (+ 1114us) server_negotiation.cc:407] Connection header received
0618 22:27:54.860177 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:54.860180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:54.860241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:54.860317 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:54.861271 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.861855 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.863329 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.863560 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.863641 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.864871 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:54.864875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:54.864878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:54.864894 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:54.864918 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:54.864928 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:54.865002 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:54.865145 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 22:27:54.865210 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:54.866254 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:54.857874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35299 (local address 127.16.243.130:59953)
0618 22:27:54.858541 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:54.858619 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:54.859952 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:54.860501 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:54.860509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:54.860600 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:54.861133 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.861147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.861989 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.861993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.863216 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:54.863228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:54.863717 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:54.863721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:54.863917 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:54.864553 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:54.864555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:54.864623 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:54.864712 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:54.865012 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:54.865020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:54.865025 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:54.865134 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 22:27:54.865382 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260618 22:27:54.867436 26925 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35299
I20260618 22:27:54.867676 26925 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:54.868207 26925 heartbeater.cc:511] Master 127.16.243.190:35299 requested a full tablet report, sending...
I20260618 22:27:54.869230 26604 ts_manager.cc:194] Registered new tserver with Master: 0f64a65258974fb79f92599436bbd3c2 (127.16.243.130:34923)
I20260618 22:27:54.869804 26604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.130:59953
W20260618 22:27:54.973903 26928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:54.974197 26928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:54.974251 26928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:54.978477 26928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:54.978585 26928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:54.978667 26928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:54.983633 26928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:54.984743 26928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:54.986129 26928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:54.995849 26934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:54.995854 26933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.995972 26928 server_base.cc:1034] running on GCE node
W20260618 22:27:54.995846 26936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:54.996557 26928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:54.997262 26928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:54.998422 26928 hybrid_clock.cc:648] HybridClock initialized: now 1781821674998398 us; error 38 us; skew 500 ppm
I20260618 22:27:55.000353 26928 webserver.cc:460] Webserver started at http://127.16.243.131:42171/ using document root <none> and password file <none>
I20260618 22:27:55.000924 26928 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:55.000988 26928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:55.001196 26928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:55.003585 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "7755040a74a5492d881566c8cdacb53a"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "92a454a0c638439a9e7dbc58e87b8efa"
server_key_iv: "90a11a236e122ca80422cb1848775a18"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.004185 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "7755040a74a5492d881566c8cdacb53a"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "92a454a0c638439a9e7dbc58e87b8efa"
server_key_iv: "90a11a236e122ca80422cb1848775a18"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.007964 26928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:55.010682 26942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.012080 26928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:55.012202 26928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "7755040a74a5492d881566c8cdacb53a"
format_stamp: "Formatted at 2026-06-18 22:27:54 on dist-test-slave-jm5g"
server_key: "92a454a0c638439a9e7dbc58e87b8efa"
server_key_iv: "90a11a236e122ca80422cb1848775a18"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.012329 26928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:55.024420 26928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:55.025120 26928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:55.025288 26928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:55.025861 26928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:55.026870 26928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:55.026917 26928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.026959 26928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:55.026988 26928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.035602 26928 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:45925
I20260618 22:27:55.035632 27055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:45925 every 8 connection(s)
I20260618 22:27:55.036511 26928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:55.039827 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 26928
I20260618 22:27:55.039927 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:55.040148 17358 external_mini_cluster.cc:1428] Setting key b88e7e8aec1269b0b4579672c251a4d0
I20260618 22:27:55.045984 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.038316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:46759 (local address 127.16.243.190:35299)
0618 22:27:55.038454 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.038458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.039585 (+ 1127us) server_negotiation.cc:407] Connection header received
0618 22:27:55.041157 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.041160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.041216 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.041328 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.042412 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:55.043005 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:55.044152 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:55.044359 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:55.044427 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:55.045539 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:55.045544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:55.045547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:55.045563 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:55.045587 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:55.045596 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:55.045676 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:55.045801 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:27:55.045856 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:27:55.046963 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.038759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35299 (local address 127.16.243.131:46759)
0618 22:27:55.039423 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:55.039465 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:55.040895 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:55.041566 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:55.041575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:55.041679 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:55.042248 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:55.042267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:55.043117 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:55.043121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:55.044044 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:55.044055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:55.044468 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:55.044471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:55.044627 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:55.045227 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:55.045228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:55.045273 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:55.045343 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:55.045668 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:55.045674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:55.045679 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:55.045801 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 22:27:55.046079 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
I20260618 22:27:55.048121 27056 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35299
I20260618 22:27:55.048352 27056 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:55.048866 27056 heartbeater.cc:511] Master 127.16.243.190:35299 requested a full tablet report, sending...
I20260618 22:27:55.049908 26604 ts_manager.cc:194] Registered new tserver with Master: 7755040a74a5492d881566c8cdacb53a (127.16.243.131:45925)
I20260618 22:27:55.050450 26604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.131:46759
I20260618 22:27:55.053977 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:27:55.056730 26645 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:55.055799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.16.243.190:35299)
0618 22:27:55.055966 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.055971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.056107 (+ 136us) server_negotiation.cc:407] Connection header received
0618 22:27:55.056222 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.056225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.056269 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.056332 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.056595 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54820: BlockingRecv error: recv got EOF from 127.0.0.1:54820 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:27:55.056829 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26666
I20260618 22:27:55.061775 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26797
I20260618 22:27:55.066041 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26928
I20260618 22:27:55.070132 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 26575
2026-06-18T22:27:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (857 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-18T22:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:55Z Disabled control of system clock
I20260618 22:27:55.096743 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44437
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:44437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:27:55.206477 27067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:55.206835 27067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:55.206874 27067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:55.210945 27067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:55.211012 27067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:55.211026 27067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:55.211042 27067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:55.211056 27067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:55.215960 27067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:44437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:55.217020 27067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:55.218369 27067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:55.226845 27067 server_base.cc:1034] running on GCE node
W20260618 22:27:55.226796 27072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:55.226797 27075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:55.226814 27073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.227663 27067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:55.228858 27067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:55.230029 27067 hybrid_clock.cc:648] HybridClock initialized: now 1781821675230002 us; error 34 us; skew 500 ppm
I20260618 22:27:55.231904 27067 webserver.cc:460] Webserver started at http://127.16.243.190:42447/ using document root <none> and password file <none>
I20260618 22:27:55.232472 27067 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:55.232538 27067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:55.232728 27067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:55.234829 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "b9a9dd3db6054450a15aa5d5ef770ce2"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "e744379b3564924c617e20eeaa9de520"
server_key_iv: "db419bcb1e739ee9ecd15ec5e1acd31e"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.235374 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "b9a9dd3db6054450a15aa5d5ef770ce2"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "e744379b3564924c617e20eeaa9de520"
server_key_iv: "db419bcb1e739ee9ecd15ec5e1acd31e"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.239106 27067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:27:55.241668 27081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.243033 27067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:27:55.243152 27067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "b9a9dd3db6054450a15aa5d5ef770ce2"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "e744379b3564924c617e20eeaa9de520"
server_key_iv: "db419bcb1e739ee9ecd15ec5e1acd31e"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.243260 27067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:55.247932 27067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:55.248575 27067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:55.248734 27067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:55.255381 27067 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:44437
I20260618 22:27:55.255405 27133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:44437 every 8 connection(s)
I20260618 22:27:55.256337 27067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:55.259660 27134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:55.262113 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27067
I20260618 22:27:55.262220 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:55.262495 17358 external_mini_cluster.cc:1428] Setting key cd6e1db11f4eb8664b540ac480b7cf0a
I20260618 22:27:55.266515 27134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2: Bootstrap starting.
I20260618 22:27:55.269003 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.264393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.16.243.190:44437)
0618 22:27:55.265103 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.265123 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.265168 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:27:55.265985 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.266008 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.266470 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.266869 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.267533 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:55.267538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:55.267553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:55.267565 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:55.267626 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:55.267648 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:55.267791 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:55.267858 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 22:27:55.268105 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260618 22:27:55.269253 27134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:55.269994 27134 log.cc:824] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:55.272002 27134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2: No bootstrap required, opened a new log
I20260618 22:27:55.275442 27134 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } }
I20260618 22:27:55.275679 27134 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:55.275744 27134 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a9dd3db6054450a15aa5d5ef770ce2, State: Initialized, Role: FOLLOWER
I20260618 22:27:55.276191 27134 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } }
I20260618 22:27:55.276355 27134 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:55.276424 27134 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:55.276522 27134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:55.277436 27134 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } }
I20260618 22:27:55.277767 27134 leader_election.cc:302] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a9dd3db6054450a15aa5d5ef770ce2; no voters:
I20260618 22:27:55.278183 27134 leader_election.cc:288] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:55.278285 27139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:55.278532 27139 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [term 1 LEADER]: Becoming Leader. State: Replica: b9a9dd3db6054450a15aa5d5ef770ce2, State: Running, Role: LEADER
I20260618 22:27:55.278889 27139 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } }
I20260618 22:27:55.279481 27134 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:55.280463 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a9dd3db6054450a15aa5d5ef770ce2. Latest consensus state: current_term: 1 leader_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } } }
I20260618 22:27:55.280591 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:55.280879 27140 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a9dd3db6054450a15aa5d5ef770ce2" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44437 } } }
I20260618 22:27:55.280982 27140 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:55.281064 27148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:55.283864 27148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:55.289120 27148 catalog_manager.cc:1347] Generated new cluster ID: 08b02d39c6794f1082fdacb390016d79
I20260618 22:27:55.289206 27148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:55.303791 27148 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:55.304157 27148 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:55.313506 27148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a9dd3db6054450a15aa5d5ef770ce2: Generated new TSK 0
I20260618 22:27:55.314234 27148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:27:55.327059 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--builtin_ntp_servers=127.16.243.148:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:27:55.437292 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:55.437563 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:55.437603 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:55.441627 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:55.441694 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:55.441752 27158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:55.446637 27158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:55.447731 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:55.449091 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:55.458168 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.458333 27158 server_base.cc:1034] running on GCE node
W20260618 22:27:55.458168 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:55.458226 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.458858 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:55.459483 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:55.460654 27158 hybrid_clock.cc:648] HybridClock initialized: now 1781821675460630 us; error 33 us; skew 500 ppm
I20260618 22:27:55.462502 27158 webserver.cc:460] Webserver started at http://127.16.243.129:45147/ using document root <none> and password file <none>
I20260618 22:27:55.463156 27158 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:55.463225 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:55.463425 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:55.465525 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "52f939f657ef428395bc3d0829f16642"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "4cdf231196012e0a158819fb0de708c8"
server_key_iv: "3498b3da8edcaa88ebe1965a380c0a70"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.466074 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "52f939f657ef428395bc3d0829f16642"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "4cdf231196012e0a158819fb0de708c8"
server_key_iv: "3498b3da8edcaa88ebe1965a380c0a70"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.469827 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:27:55.472467 27172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.473702 27158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:55.473816 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "52f939f657ef428395bc3d0829f16642"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "4cdf231196012e0a158819fb0de708c8"
server_key_iv: "3498b3da8edcaa88ebe1965a380c0a70"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.473923 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:55.478451 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:55.479082 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:55.479234 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:55.479755 27158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:55.480669 27158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:55.480713 27158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.480763 27158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:55.480792 27158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.488965 27158 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:45477
I20260618 22:27:55.489006 27285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:45477 every 8 connection(s)
I20260618 22:27:55.489861 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:27:55.492960 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27158
I20260618 22:27:55.493085 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:55.493405 17358 external_mini_cluster.cc:1428] Setting key 66f5093bbc2b04203fa233d127cd22e2
I20260618 22:27:55.495456 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--builtin_ntp_servers=127.16.243.148:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260618 22:27:55.496296 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.491698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:54611 (local address 127.16.243.190:44437)
0618 22:27:55.491826 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.491830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.493052 (+ 1222us) server_negotiation.cc:407] Connection header received
0618 22:27:55.494617 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.494620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.494661 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.494723 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.495904 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:55.495907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:55.495909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:55.495920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:55.495951 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:55.495959 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:55.496032 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:55.496160 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 22:27:55.496196 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:27:55.497458 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.492120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:44437 (local address 127.16.243.129:54611)
0618 22:27:55.492861 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:55.492924 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:55.494399 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:55.494903 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:55.494911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:55.495031 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:55.495609 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:55.495611 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:55.495671 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:55.495755 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:55.496035 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:55.496041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:55.496047 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:27:55.496212 (+ 165us) client_negotiation.cc:240] Negotiation successful
0618 22:27:55.496471 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260618 22:27:55.498699 27286 heartbeater.cc:347] Connected to a master server at 127.16.243.190:44437
I20260618 22:27:55.498950 27286 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:55.499372 27286 heartbeater.cc:511] Master 127.16.243.190:44437 requested a full tablet report, sending...
I20260618 22:27:55.500934 27098 ts_manager.cc:194] Registered new tserver with Master: 52f939f657ef428395bc3d0829f16642 (127.16.243.129:45477)
W20260618 22:27:55.603559 27289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:55.603842 27289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:55.603894 27289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:55.608001 27289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:55.608073 27289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:55.608150 27289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:55.613029 27289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:55.614138 27289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:55.615523 27289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:55.625056 27294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:55.625073 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.625314 27289 server_base.cc:1034] running on GCE node
W20260618 22:27:55.625059 27295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.625957 27289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:55.626621 27289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:55.627810 27289 hybrid_clock.cc:648] HybridClock initialized: now 1781821675627776 us; error 45 us; skew 500 ppm
I20260618 22:27:55.629666 27289 webserver.cc:460] Webserver started at http://127.16.243.130:44033/ using document root <none> and password file <none>
I20260618 22:27:55.630218 27289 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:55.630280 27289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:55.630479 27289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:55.632579 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "e928344f270d4511a8b3bec09cdc6520"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "050c3a3bd4e86afac4352d6763465bb9"
server_key_iv: "a269ab4f99b41b5f94015c3a28561afd"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.633109 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "e928344f270d4511a8b3bec09cdc6520"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "050c3a3bd4e86afac4352d6763465bb9"
server_key_iv: "a269ab4f99b41b5f94015c3a28561afd"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.636685 27289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:27:55.639155 27303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.640334 27289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:27:55.640441 27289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "e928344f270d4511a8b3bec09cdc6520"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "050c3a3bd4e86afac4352d6763465bb9"
server_key_iv: "a269ab4f99b41b5f94015c3a28561afd"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.640543 27289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:55.645146 27289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:55.645735 27289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:55.645915 27289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:55.646476 27289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:55.647411 27289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:55.647456 27289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.647506 27289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:55.647535 27289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.655699 27289 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:35617
I20260618 22:27:55.655728 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:35617 every 8 connection(s)
I20260618 22:27:55.656610 27289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:55.661443 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27289
I20260618 22:27:55.661554 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:55.661808 17358 external_mini_cluster.cc:1428] Setting key 2f261011fec240d0ee1f074d496c7193
I20260618 22:27:55.662758 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.658523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:48509 (local address 127.16.243.190:44437)
0618 22:27:55.658689 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.658694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.659776 (+ 1082us) server_negotiation.cc:407] Connection header received
0618 22:27:55.661019 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.661022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.661064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.661156 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.662308 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:55.662311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:55.662314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:55.662328 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:55.662347 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:55.662355 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:55.662430 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:55.662585 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 22:27:55.662628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:27:55.663677 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--builtin_ntp_servers=127.16.243.148:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260618 22:27:55.663692 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.658983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:44437 (local address 127.16.243.130:48509)
0618 22:27:55.659647 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:55.659689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:55.660836 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:55.661389 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:55.661396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:55.661491 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:55.662044 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:55.662045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:55.662088 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:55.662159 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:55.662424 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:55.662432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:55.662437 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:27:55.662618 (+ 181us) client_negotiation.cc:240] Negotiation successful
0618 22:27:55.662868 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260618 22:27:55.664785 27417 heartbeater.cc:347] Connected to a master server at 127.16.243.190:44437
I20260618 22:27:55.665014 27417 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:55.665400 27417 heartbeater.cc:511] Master 127.16.243.190:44437 requested a full tablet report, sending...
I20260618 22:27:55.666381 27098 ts_manager.cc:194] Registered new tserver with Master: e928344f270d4511a8b3bec09cdc6520 (127.16.243.130:35617)
W20260618 22:27:55.772863 27420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:55.773149 27420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:55.773206 27420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:55.777360 27420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:55.777429 27420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:55.777506 27420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:55.782632 27420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:55.783758 27420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:55.785198 27420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:55.794632 27426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.794682 27420 server_base.cc:1034] running on GCE node
W20260618 22:27:55.794636 27425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:55.794639 27428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:55.795246 27420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:55.795871 27420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:55.797025 27420 hybrid_clock.cc:648] HybridClock initialized: now 1781821675797004 us; error 36 us; skew 500 ppm
I20260618 22:27:55.798832 27420 webserver.cc:460] Webserver started at http://127.16.243.131:46129/ using document root <none> and password file <none>
I20260618 22:27:55.799401 27420 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:55.799465 27420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:55.799649 27420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:55.801687 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "adff287b5fc349f1ae88334b288b25eb"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "5ab79a975a419414f26e2cdbf6adc3e5"
server_key_iv: "ae48f7f3b896882256342a3f0181ee79"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.802237 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "adff287b5fc349f1ae88334b288b25eb"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "5ab79a975a419414f26e2cdbf6adc3e5"
server_key_iv: "ae48f7f3b896882256342a3f0181ee79"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.805905 27420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:55.808527 27434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.809743 27420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:27:55.809851 27420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "adff287b5fc349f1ae88334b288b25eb"
format_stamp: "Formatted at 2026-06-18 22:27:55 on dist-test-slave-jm5g"
server_key: "5ab79a975a419414f26e2cdbf6adc3e5"
server_key_iv: "ae48f7f3b896882256342a3f0181ee79"
server_key_version: "encryptionkey@0"
I20260618 22:27:55.809957 27420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:55.814759 27420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:55.815421 27420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:55.815587 27420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:55.816156 27420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:55.817091 27420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:55.817137 27420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.817188 27420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:55.817214 27420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:55.825681 27420 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35657
I20260618 22:27:55.825744 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35657 every 8 connection(s)
I20260618 22:27:55.826670 27420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:27:55.829560 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27420
I20260618 22:27:55.829675 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:55.830003 17358 external_mini_cluster.cc:1428] Setting key 709db0bd706bbe3ed84406f1dc87e9cf
I20260618 22:27:55.833001 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.828459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:40427 (local address 127.16.243.190:44437)
0618 22:27:55.828606 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.828610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.829845 (+ 1235us) server_negotiation.cc:407] Connection header received
0618 22:27:55.831449 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.831452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.831494 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.831581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.832667 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:55.832670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:55.832674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:27:55.832685 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:55.832706 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:27:55.832714 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:55.832782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:55.832867 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 22:27:55.832901 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:27:55.833909 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:55.828863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:44437 (local address 127.16.243.131:40427)
0618 22:27:55.829672 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:55.829732 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:55.831217 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:55.831720 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:55.831728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:55.831822 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:55.832397 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:27:55.832399 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:55.832464 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:27:55.832536 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:55.832776 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:55.832781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:55.832784 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:27:55.832908 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 22:27:55.833133 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260618 22:27:55.834933 27548 heartbeater.cc:347] Connected to a master server at 127.16.243.190:44437
I20260618 22:27:55.835150 27548 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:55.835561 27548 heartbeater.cc:511] Master 127.16.243.190:44437 requested a full tablet report, sending...
I20260618 22:27:55.836493 27098 ts_manager.cc:194] Registered new tserver with Master: adff287b5fc349f1ae88334b288b25eb (127.16.243.131:35657)
I20260618 22:27:55.844105 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:27:55.846939 27137 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:27:55.846061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.16.243.190:44437)
0618 22:27:55.846228 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:55.846233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:55.846317 (+ 84us) server_negotiation.cc:407] Connection header received
0618 22:27:55.846474 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:55.846477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:55.846515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:55.846613 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:55.846805 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56436: BlockingRecv error: recv got EOF from 127.0.0.1:56436 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:27:55.847105 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27158
I20260618 22:27:55.851666 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27289
I20260618 22:27:55.856552 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27420
I20260618 22:27:55.862665 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27067
2026-06-18T22:27:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:55 dist-test-slave-jm5g krb5kdc[27558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:55 dist-test-slave-jm5g krb5kdc[27558](info): set up 2 sockets
Jun 18 22:27:55 dist-test-slave-jm5g krb5kdc[27558](info): commencing operation
krb5kdc: starting...
W20260618 22:27:56.815234 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:56 dist-test-slave-jm5g krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:27:56.927040 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34235
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:34235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:57.035422 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:57.035710 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:57.035751 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:57.039783 27571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:27:57.039851 27571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:57.039870 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:57.039896 27571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:27:57.039924 27571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:27:57.045445 27571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:34235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:57.046581 27571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:57.047609 27571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:57.055810 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.055862 27571 server_base.cc:1034] running on GCE node
W20260618 22:27:57.055805 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.055835 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.056685 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:57.057787 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:57.058944 27571 hybrid_clock.cc:648] HybridClock initialized: now 1781821677058906 us; error 50 us; skew 500 ppm
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:57.061692 27571 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:27:57.062602 27571 webserver.cc:460] Webserver started at http://127.16.243.190:42819/ using document root <none> and password file <none>
I20260618 22:27:57.063144 27571 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:57.063200 27571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:57.063370 27571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:57.065478 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "f63c839e3a5a42ec83cc8a9c9469b843"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "8083a30c3055699032fd815c4b679394"
server_key_iv: "b1d2b7b120a9a9936df496c50296bb2e"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.066039 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "f63c839e3a5a42ec83cc8a9c9469b843"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "8083a30c3055699032fd815c4b679394"
server_key_iv: "b1d2b7b120a9a9936df496c50296bb2e"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.069747 27571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:57.072221 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.073494 27571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:57.073594 27571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "f63c839e3a5a42ec83cc8a9c9469b843"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "8083a30c3055699032fd815c4b679394"
server_key_iv: "b1d2b7b120a9a9936df496c50296bb2e"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.073704 27571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:57.088613 27571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:57.091802 27571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:57.091976 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:57.098536 27571 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:34235
I20260618 22:27:57.098836 27638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:34235 every 8 connection(s)
I20260618 22:27:57.099561 27571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:27:57.102416 27639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:57.103116 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27571
I20260618 22:27:57.103211 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:27:57.103466 17358 external_mini_cluster.cc:1428] Setting key aaa989261a7f43ba18d7ab76614db9be
I20260618 22:27:57.109061 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: Bootstrap starting.
I20260618 22:27:57.111569 27639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:57.112296 27639 log.cc:824] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:57.114588 27639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: No bootstrap required, opened a new log
I20260618 22:27:57.118156 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.104884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.16.243.190:34235)
0618 22:27:57.105310 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.105329 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.105374 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:27:57.106169 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.106192 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.106664 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.107022 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:57.107635 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.109065 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.109943 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.110664 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.110820 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.113161 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:57.113168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:57.113185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:57.113214 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:57.115082 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.115521 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.115527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.115533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.115675 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.115913 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.115916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.115917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.116428 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:57.116626 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.116841 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.117172 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260618 22:27:57.119155 27639 raft_consensus.cc:348] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } }
I20260618 22:27:57.119439 27639 raft_consensus.cc:374] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:57.119554 27639 raft_consensus.cc:729] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63c839e3a5a42ec83cc8a9c9469b843, State: Initialized, Role: FOLLOWER
I20260618 22:27:57.120011 27639 consensus_queue.cc:260] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } }
I20260618 22:27:57.120146 27639 raft_consensus.cc:388] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:27:57.120209 27639 raft_consensus.cc:482] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:27:57.120306 27639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:57.121327 27639 raft_consensus.cc:504] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } }
I20260618 22:27:57.121663 27639 leader_election.cc:302] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f63c839e3a5a42ec83cc8a9c9469b843; no voters:
I20260618 22:27:57.121973 27639 leader_election.cc:288] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:27:57.122067 27644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:57.122267 27644 raft_consensus.cc:686] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [term 1 LEADER]: Becoming Leader. State: Replica: f63c839e3a5a42ec83cc8a9c9469b843, State: Running, Role: LEADER
I20260618 22:27:57.122639 27644 consensus_queue.cc:237] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } }
I20260618 22:27:57.123142 27639 sys_catalog.cc:564] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:27:57.124294 27646 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f63c839e3a5a42ec83cc8a9c9469b843. Latest consensus state: current_term: 1 leader_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } } }
I20260618 22:27:57.124300 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63c839e3a5a42ec83cc8a9c9469b843" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34235 } } }
I20260618 22:27:57.124419 27646 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [sys.catalog]: This master's current role is: LEADER
I20260618 22:27:57.124420 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843 [sys.catalog]: This master's current role is: LEADER
W20260618 22:27:57.127846 27659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:27:57.127928 27659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:27:57.128141 27654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:27:57.129021 27654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:27:57.134532 27654 catalog_manager.cc:1347] Generated new cluster ID: 988cfd247edd4a7c8e9608e84b469709
I20260618 22:27:57.134634 27654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:27:57.156857 27654 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:27:57.157827 27654 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:27:57.164729 27654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: Generated new TSK 0
I20260618 22:27:57.165597 27654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:27:57.195858 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:57.308705 27665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:57.309000 27665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:57.309046 27665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:57.313180 27665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:57.313251 27665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:57.313325 27665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:27:57.318696 27665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:57.319770 27665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:57.320775 27665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:57.330538 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.330533 27673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.330538 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.330606 27665 server_base.cc:1034] running on GCE node
I20260618 22:27:57.331262 27665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:57.331915 27665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:57.333072 27665 hybrid_clock.cc:648] HybridClock initialized: now 1781821677333053 us; error 37 us; skew 500 ppm
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:57.335726 27665 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:27:57.336640 27665 webserver.cc:460] Webserver started at http://127.16.243.129:33675/ using document root <none> and password file <none>
I20260618 22:27:57.337175 27665 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:57.337240 27665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:57.337433 27665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:57.339551 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "8cfc2b25b03248f1827f2b2e21f8175a"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "bd25d3e3d10f174587a2d2959d6dc636"
server_key_iv: "816843b25dc448dce5061d30a28228fb"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.340085 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "8cfc2b25b03248f1827f2b2e21f8175a"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "bd25d3e3d10f174587a2d2959d6dc636"
server_key_iv: "816843b25dc448dce5061d30a28228fb"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.343832 27665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:27:57.346283 27680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.347570 27665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:57.347719 27665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "8cfc2b25b03248f1827f2b2e21f8175a"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "bd25d3e3d10f174587a2d2959d6dc636"
server_key_iv: "816843b25dc448dce5061d30a28228fb"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.347827 27665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:57.378363 27665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:57.381389 27665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:57.381577 27665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:57.382164 27665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:57.383021 27665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:57.383067 27665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.383121 27665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:57.383150 27665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.391494 27665 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:45735
I20260618 22:27:57.391573 27793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:45735 every 8 connection(s)
I20260618 22:27:57.392390 27665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:57.402634 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27665
I20260618 22:27:57.402757 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:27:57.403036 17358 external_mini_cluster.cc:1428] Setting key 970ff9c9fb253d6fad88f8bfb747ec1c
I20260618 22:27:57.405669 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.394130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:50781 (local address 127.16.243.190:34235)
0618 22:27:57.394259 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.394262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.395081 (+ 819us) server_negotiation.cc:407] Connection header received
0618 22:27:57.396424 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.396427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.396485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.396559 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:57.397719 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.398263 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.399410 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.399647 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.399732 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.402123 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:57.402126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:57.402128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:57.402153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:57.403609 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.404153 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.404156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.404157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.404240 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.404615 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.404619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.404620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.404836 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:57.404935 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.405385 (+ 450us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.405515 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:57.406471 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.394486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34235 (local address 127.16.243.129:50781)
0618 22:27:57.394931 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.394975 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.396214 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.396728 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.396735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.397086 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:57.397561 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.397574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.398382 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.398385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.399287 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.399296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.399788 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.399792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.399947 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.400471 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:57.400486 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:57.401941 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:57.403713 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.403719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.403737 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.404055 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.404336 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.404339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.404341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.404523 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.404923 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:57.404927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:57.405171 (+ 244us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.405327 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.405586 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260618 22:27:57.407764 27794 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34235
I20260618 22:27:57.408017 27794 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:57.408622 27794 heartbeater.cc:511] Master 127.16.243.190:34235 requested a full tablet report, sending...
I20260618 22:27:57.410178 27603 ts_manager.cc:194] Registered new tserver with Master: 8cfc2b25b03248f1827f2b2e21f8175a (127.16.243.129:45735)
I20260618 22:27:57.411356 27603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:50781
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:27:57.429154 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:57.538420 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:57.538749 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:57.538820 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:57.542948 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:57.543028 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:57.543118 27799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:27:57.548897 27799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:57.550031 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:57.551106 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:27:57.560527 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.560642 27799 server_base.cc:1034] running on GCE node
W20260618 22:27:57.560523 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.560527 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.561136 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:57.561793 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:57.562971 27799 hybrid_clock.cc:648] HybridClock initialized: now 1781821677562948 us; error 35 us; skew 500 ppm
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:57.565701 27799 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:27:57.566659 27799 webserver.cc:460] Webserver started at http://127.16.243.130:43991/ using document root <none> and password file <none>
I20260618 22:27:57.567214 27799 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:57.567274 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:57.567473 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:57.569483 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "df8059965fbf44d7b74f5b21996f305d"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "3397cb1ca17e42dca40b169914bda5a4"
server_key_iv: "aa4c0cd32dad4ed76a58871121914f63"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.570010 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "df8059965fbf44d7b74f5b21996f305d"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "3397cb1ca17e42dca40b169914bda5a4"
server_key_iv: "aa4c0cd32dad4ed76a58871121914f63"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.573693 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:27:57.576241 27814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.577569 27799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:27:57.577697 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "df8059965fbf44d7b74f5b21996f305d"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "3397cb1ca17e42dca40b169914bda5a4"
server_key_iv: "aa4c0cd32dad4ed76a58871121914f63"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.577804 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:57.604238 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:57.607236 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:57.607417 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:57.608000 27799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:57.608810 27799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:57.608857 27799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.608908 27799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:57.608934 27799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.617419 27799 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:41791
I20260618 22:27:57.617450 27927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:41791 every 8 connection(s)
I20260618 22:27:57.618302 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:27:57.625367 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27799
I20260618 22:27:57.625478 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:27:57.625751 17358 external_mini_cluster.cc:1428] Setting key 19bde1368b5468f68e213cb33e978f8e
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:57.632611 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.620069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:37449 (local address 127.16.243.190:34235)
0618 22:27:57.620195 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.620199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.621037 (+ 838us) server_negotiation.cc:407] Connection header received
0618 22:27:57.622350 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.622353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.622402 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.622496 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:57.623809 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.624495 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.625921 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.626168 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.626277 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.629031 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:57.629034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:57.629036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:57.629061 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:57.630341 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.630992 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.630994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.630996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.631072 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.631433 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.631435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.631437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.631628 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:57.631699 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.632366 (+ 667us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.632471 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:27:57.633857 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.620462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34235 (local address 127.16.243.130:37449)
0618 22:27:57.620888 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.620931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.622149 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.622716 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.622724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.623121 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:57.623625 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.623639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.624637 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.624639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.625804 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.625816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.626348 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.626352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.626538 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.627203 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:57.627224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:57.628846 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:57.630483 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.630490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.630508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.630885 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.631158 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.631161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.631163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.631352 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.631727 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:57.631735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:57.632104 (+ 369us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.632337 (+ 233us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.632721 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:27:57.635138 27928 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34235
I20260618 22:27:57.635407 27928 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:57.635915 27928 heartbeater.cc:511] Master 127.16.243.190:34235 requested a full tablet report, sending...
I20260618 22:27:57.636940 27603 ts_manager.cc:194] Registered new tserver with Master: df8059965fbf44d7b74f5b21996f305d (127.16.243.130:41791)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:27:57.637576 27603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:37449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:27:57.650693 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:27:57.761246 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:27:57.761547 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:27:57.761626 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:27:57.765745 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:27:57.765825 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:27:57.765915 27933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:27:57.771337 27933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:27:57.772418 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:27:57.773432 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:27:57.782829 27933 server_base.cc:1034] running on GCE node
W20260618 22:27:57.782778 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.782778 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:27:57.782797 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:27:57.783519 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:27:57.784137 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:27:57.785303 27933 hybrid_clock.cc:648] HybridClock initialized: now 1781821677785282 us; error 31 us; skew 500 ppm
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:27:57.787926 27933 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:27:57.788756 27933 webserver.cc:460] Webserver started at http://127.16.243.131:41787/ using document root <none> and password file <none>
I20260618 22:27:57.789260 27933 fs_manager.cc:359] Metadata directory not provided
I20260618 22:27:57.789315 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:27:57.789475 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:27:57.791447 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "8b804046ed3d477896edb928a5dd2ab8"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "27f7afd6c5a0811b76189b84d7d09054"
server_key_iv: "c0603ba780469b6c7b1d4eb4ec1a348a"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.791951 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "8b804046ed3d477896edb928a5dd2ab8"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "27f7afd6c5a0811b76189b84d7d09054"
server_key_iv: "c0603ba780469b6c7b1d4eb4ec1a348a"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.795534 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:27:57.797853 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.799099 27933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:27:57.799201 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "8b804046ed3d477896edb928a5dd2ab8"
format_stamp: "Formatted at 2026-06-18 22:27:57 on dist-test-slave-jm5g"
server_key: "27f7afd6c5a0811b76189b84d7d09054"
server_key_iv: "c0603ba780469b6c7b1d4eb4ec1a348a"
server_key_version: "encryptionkey@0"
I20260618 22:27:57.799299 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:27:57.822430 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:27:57.825537 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:27:57.825713 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:27:57.826296 27933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:27:57.827165 27933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:27:57.827212 27933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.827252 27933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:27:57.827281 27933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:27:57.835659 27933 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:44537
I20260618 22:27:57.835690 28061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:44537 every 8 connection(s)
I20260618 22:27:57.836551 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
Jun 18 22:27:57 dist-test-slave-jm5g krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821677, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:27:57.846532 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 27933
I20260618 22:27:57.846678 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:27:57.846931 17358 external_mini_cluster.cc:1428] Setting key 0ddd85fcef8aab315c32b1aefdfaba7e
I20260618 22:27:57.849773 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.838318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:47001 (local address 127.16.243.190:34235)
0618 22:27:57.838454 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.838457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.839316 (+ 859us) server_negotiation.cc:407] Connection header received
0618 22:27:57.840689 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.840692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.840750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.840822 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:57.842026 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.842587 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.843714 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.843937 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.844023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.846412 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:57.846415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:57.846418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:57.846443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:57.847869 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.848349 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.848353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.848355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.848433 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.848731 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.848734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.848736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.848925 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:57.849007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.849503 (+ 496us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.849620 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:27:57.850660 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.838724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:34235 (local address 127.16.243.131:47001)
0618 22:27:57.839171 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.839214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.840477 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.841015 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.841023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.841386 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:27:57.841875 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.841888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.842698 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.842701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.843600 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.843609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.844074 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.844078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.844278 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.844815 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:27:57.844830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:27:57.846245 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:27:57.847985 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.847992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.848009 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.848260 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.848509 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:27:57.848512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:27:57.848514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:27:57.848651 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:27:57.849017 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:27:57.849024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:27:57.849277 (+ 253us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.849456 (+ 179us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.849792 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260618 22:27:57.851732 28062 heartbeater.cc:347] Connected to a master server at 127.16.243.190:34235
I20260618 22:27:57.851963 28062 heartbeater.cc:464] Registering TS with master...
I20260618 22:27:57.852434 28062 heartbeater.cc:511] Master 127.16.243.190:34235 requested a full tablet report, sending...
I20260618 22:27:57.853434 27603 ts_manager.cc:194] Registered new tserver with Master: 8b804046ed3d477896edb928a5dd2ab8 (127.16.243.131:44537)
I20260618 22:27:57.853981 27603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:47001
I20260618 22:27:57.860469 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:27:57.871008 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.862305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.16.243.190:34235)
0618 22:27:57.862438 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.862442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.862652 (+ 210us) server_negotiation.cc:407] Connection header received
0618 22:27:57.862758 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.862761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.862810 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.862902 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:27:57.863541 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.864165 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.865055 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.865277 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.865363 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.867583 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:27:57.867587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:27:57.867589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:27:57.867620 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:27:57.869330 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.869813 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.869816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.869818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.869907 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:27:57.870137 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:27:57.870139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:27:57.870140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:27:57.870355 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:27:57.870484 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.870772 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.870875 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:27:57.874095 27603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:27:57.877205 27603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:27:57.891939 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.885881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45735 (local address 127.0.0.1:51186)
0618 22:27:57.887398 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.887411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.887495 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.887957 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.887960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.887976 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:57.888071 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.888077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.889093 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.889096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.890613 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.890619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891399 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.891516 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891554 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.891661 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.891789 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":96,"thread_start_us":124,"threads_started":1}
I20260618 22:27:57.891958 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.886647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.16.243.129:45735)
0618 22:27:57.887013 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.887027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.887585 (+ 558us) server_negotiation.cc:407] Connection header received
0618 22:27:57.887664 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.887669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.887837 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.887964 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:57.888189 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.888972 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.890727 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891288 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891541 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.891707 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.891801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":67,"threads_started":1}
I20260618 22:27:57.892242 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.886606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44537 (local address 127.0.0.1:35574)
0618 22:27:57.887497 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.887509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.887637 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.888118 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.888121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.888137 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:57.888229 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.888235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.889502 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.889505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.890663 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.890675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891817 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.891954 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891971 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.892027 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.892109 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":716,"thread_start_us":75,"threads_started":1}
I20260618 22:27:57.892325 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.885971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.16.243.130:41791)
0618 22:27:57.886284 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.886294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.886940 (+ 646us) server_negotiation.cc:407] Connection header received
0618 22:27:57.887141 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.887146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.887345 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.887518 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:57.888003 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.889190 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.890693 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891516 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891639 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891738 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.892099 (+ 361us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.892190 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
I20260618 22:27:57.892578 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.886859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.16.243.131:44537)
0618 22:27:57.887282 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.887295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.887578 (+ 283us) server_negotiation.cc:407] Connection header received
0618 22:27:57.887754 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.887759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.887965 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.888101 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:57.888316 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.889389 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.890796 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891537 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891738 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.892338 (+ 600us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.892439 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":94,"threads_started":1}
I20260618 22:27:57.892613 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.885838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:41791 (local address 127.0.0.1:33966)
0618 22:27:57.886794 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.886844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.886993 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.887723 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.887729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.887752 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:57.887876 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.887882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.889318 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.889322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.890589 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.890601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.891800 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.891804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.891939 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.891962 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.892027 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.892504 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":456,"thread_start_us":62,"threads_started":1}
I20260618 22:27:57.895298 27862 tablet_service.cc:1467] Processing CreateTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 (DEFAULT_TABLE table=test-table [id=8b7af1b43d3f48349220f3485476b14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:57.895406 27728 tablet_service.cc:1467] Processing CreateTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 (DEFAULT_TABLE table=test-table [id=8b7af1b43d3f48349220f3485476b14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:57.895593 27996 tablet_service.cc:1467] Processing CreateTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 (DEFAULT_TABLE table=test-table [id=8b7af1b43d3f48349220f3485476b14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:27:57.896360 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e916e739ab4d9782dbb1ba42d68ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:57.896462 27728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e916e739ab4d9782dbb1ba42d68ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:57.896462 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e916e739ab4d9782dbb1ba42d68ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:27:57.902846 28079 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Bootstrap starting.
I20260618 22:27:57.905675 28079 tablet_bootstrap.cc:654] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:57.906205 28080 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: Bootstrap starting.
I20260618 22:27:57.906522 28079 log.cc:824] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:57.906864 28081 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Bootstrap starting.
I20260618 22:27:57.908360 28080 tablet_bootstrap.cc:654] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:57.908743 28079 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: No bootstrap required, opened a new log
I20260618 22:27:57.908825 28081 tablet_bootstrap.cc:654] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Neither blocks nor log segments found. Creating new log.
I20260618 22:27:57.908967 28079 ts_tablet_manager.cc:1397] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:27:57.909066 28080 log.cc:824] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:57.910017 28081 log.cc:824] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:27:57.910764 28080 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: No bootstrap required, opened a new log
I20260618 22:27:57.910948 28080 ts_tablet_manager.cc:1397] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:57.912004 28081 tablet_bootstrap.cc:492] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: No bootstrap required, opened a new log
I20260618 22:27:57.912182 28081 ts_tablet_manager.cc:1397] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:27:57.912776 28079 raft_consensus.cc:348] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.913055 28079 raft_consensus.cc:374] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:57.913120 28079 raft_consensus.cc:729] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8059965fbf44d7b74f5b21996f305d, State: Initialized, Role: FOLLOWER
I20260618 22:27:57.913583 28079 consensus_queue.cc:260] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.914295 27928 heartbeater.cc:503] Master 127.16.243.190:34235 was elected leader, sending a full tablet report...
I20260618 22:27:57.914295 28080 raft_consensus.cc:348] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.914526 28080 raft_consensus.cc:374] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:57.914595 28080 raft_consensus.cc:729] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfc2b25b03248f1827f2b2e21f8175a, State: Initialized, Role: FOLLOWER
I20260618 22:27:57.914642 28079 ts_tablet_manager.cc:1428] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:27:57.915017 28080 consensus_queue.cc:260] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.915679 27794 heartbeater.cc:503] Master 127.16.243.190:34235 was elected leader, sending a full tablet report...
I20260618 22:27:57.915719 28081 raft_consensus.cc:348] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.915953 28081 raft_consensus.cc:374] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:27:57.915956 28080 ts_tablet_manager.cc:1428] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:27:57.915998 28081 raft_consensus.cc:729] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b804046ed3d477896edb928a5dd2ab8, State: Initialized, Role: FOLLOWER
I20260618 22:27:57.916414 28081 consensus_queue.cc:260] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.917213 28081 ts_tablet_manager.cc:1428] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:27:57.917542 28062 heartbeater.cc:503] Master 127.16.243.190:34235 was elected leader, sending a full tablet report...
I20260618 22:27:57.921810 28085 raft_consensus.cc:482] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:27:57.921976 28085 raft_consensus.cc:504] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.923105 28085 leader_election.cc:288] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cfc2b25b03248f1827f2b2e21f8175a (127.16.243.129:45735), 8b804046ed3d477896edb928a5dd2ab8 (127.16.243.131:44537)
I20260618 22:27:57.926712 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.923462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:60337 (local address 127.16.243.131:44537)
0618 22:27:57.923856 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.923860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.923873 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:27:57.923915 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.923918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.923958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.924024 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:57.924277 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.924758 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.925786 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.926183 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.926283 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.926334 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.926509 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.926603 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260618 22:27:57.926728 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.923394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44537 (local address 127.16.243.130:60337)
0618 22:27:57.923743 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.923755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.923847 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.924051 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.924053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.924067 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:57.924184 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.924189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.924886 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.924889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.925673 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.925682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.926283 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.926286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.926381 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.926427 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.926487 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.926580 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
I20260618 22:27:57.926851 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.923339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45735 (local address 127.16.243.130:55891)
0618 22:27:57.923548 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.923563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:27:57.923660 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:27:57.923921 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:27:57.923924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:27:57.923942 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:27:57.924061 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.924066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.924769 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.924795 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.925788 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:27:57.925798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.926379 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.926382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:27:57.926537 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.926586 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:27:57.926657 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 22:27:57.926727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":37}
I20260618 22:27:57.926851 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.923428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55891 (local address 127.16.243.129:45735)
0618 22:27:57.923552 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.923556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.923631 (+ 75us) server_negotiation.cc:407] Connection header received
0618 22:27:57.923783 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.923785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.923826 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.923899 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:27:57.924168 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.924671 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.925898 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.926277 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.926378 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.926427 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.926670 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.926736 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:27:57.927515 27748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e916e739ab4d9782dbb1ba42d68ef7" candidate_uuid: "df8059965fbf44d7b74f5b21996f305d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" is_pre_election: true
I20260618 22:27:57.927546 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e916e739ab4d9782dbb1ba42d68ef7" candidate_uuid: "df8059965fbf44d7b74f5b21996f305d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b804046ed3d477896edb928a5dd2ab8" is_pre_election: true
I20260618 22:27:57.927798 27748 raft_consensus.cc:2454] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8059965fbf44d7b74f5b21996f305d in term 0.
I20260618 22:27:57.927810 28016 raft_consensus.cc:2454] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8059965fbf44d7b74f5b21996f305d in term 0.
I20260618 22:27:57.928248 27818 leader_election.cc:302] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b804046ed3d477896edb928a5dd2ab8, df8059965fbf44d7b74f5b21996f305d; no voters:
I20260618 22:27:57.928514 28085 raft_consensus.cc:2781] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:27:57.928593 28085 raft_consensus.cc:482] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:27:57.928637 28085 raft_consensus.cc:3037] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:57.929659 28085 raft_consensus.cc:504] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.930008 28085 leader_election.cc:288] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [CANDIDATE]: Term 1 election: Requested vote from peers 8cfc2b25b03248f1827f2b2e21f8175a (127.16.243.129:45735), 8b804046ed3d477896edb928a5dd2ab8 (127.16.243.131:44537)
I20260618 22:27:57.930419 27748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e916e739ab4d9782dbb1ba42d68ef7" candidate_uuid: "df8059965fbf44d7b74f5b21996f305d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cfc2b25b03248f1827f2b2e21f8175a"
I20260618 22:27:57.930419 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e916e739ab4d9782dbb1ba42d68ef7" candidate_uuid: "df8059965fbf44d7b74f5b21996f305d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b804046ed3d477896edb928a5dd2ab8"
I20260618 22:27:57.930594 28016 raft_consensus.cc:3037] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:57.930598 27748 raft_consensus.cc:3037] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:27:57.931859 27748 raft_consensus.cc:2454] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8059965fbf44d7b74f5b21996f305d in term 1.
I20260618 22:27:57.931859 28016 raft_consensus.cc:2454] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8059965fbf44d7b74f5b21996f305d in term 1.
I20260618 22:27:57.932164 27815 leader_election.cc:302] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cfc2b25b03248f1827f2b2e21f8175a, df8059965fbf44d7b74f5b21996f305d; no voters:
I20260618 22:27:57.932339 28085 raft_consensus.cc:2781] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:27:57.932565 28085 raft_consensus.cc:686] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 1 LEADER]: Becoming Leader. State: Replica: df8059965fbf44d7b74f5b21996f305d, State: Running, Role: LEADER
I20260618 22:27:57.932880 28085 consensus_queue.cc:237] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } }
I20260618 22:27:57.935989 27602 catalog_manager.cc:5495] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d reported cstate change: term changed from 0 to 1, leader changed from <none> to df8059965fbf44d7b74f5b21996f305d (127.16.243.130). New cstate: current_term: 1 leader_uuid: "df8059965fbf44d7b74f5b21996f305d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8059965fbf44d7b74f5b21996f305d" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 41791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 } health_report { overall_health: UNKNOWN } } }
I20260618 22:27:57.957158 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.953881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.16.243.130:41791)
0618 22:27:57.954024 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:27:57.954028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:27:57.954042 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:27:57.954115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:27:57.954117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:27:57.954172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:27:57.954263 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:27:57.954436 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.954999 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.955827 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:27:57.956043 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:27:57.956116 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:27:57.956334 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:27:57.956758 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:27:57.956849 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:27:57.956968 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 22:27:57.957019 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:27:57.963124 27748 raft_consensus.cc:1261] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a [term 1 FOLLOWER]: Refusing update from remote peer df8059965fbf44d7b74f5b21996f305d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:57.963115 28016 raft_consensus.cc:1261] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 1 FOLLOWER]: Refusing update from remote peer df8059965fbf44d7b74f5b21996f305d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:27:57.963832 28089 consensus_queue.cc:1035] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cfc2b25b03248f1827f2b2e21f8175a" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 45735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:57.964015 28085 consensus_queue.cc:1035] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b804046ed3d477896edb928a5dd2ab8" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:27:57.981500 27602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:57.981683 27602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:27:57.984179 27602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f63c839e3a5a42ec83cc8a9c9469b843: Sending DeleteTablet for 3 replicas of tablet 86e916e739ab4d9782dbb1ba42d68ef7
I20260618 22:27:57.985054 27996 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:35574
I20260618 22:27:57.985060 27728 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51186
I20260618 22:27:57.985630 28102 tablet_replica.cc:330] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: stopping tablet replica
I20260618 22:27:57.985770 27862 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e916e739ab4d9782dbb1ba42d68ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:27:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:33966
I20260618 22:27:57.985952 28103 tablet_replica.cc:330] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8cfc2b25b03248f1827f2b2e21f8175a: stopping tablet replica
I20260618 22:27:57.986192 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27665
I20260618 22:27:57.986219 28104 tablet_replica.cc:330] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: stopping tablet replica
I20260618 22:27:57.986971 28104 raft_consensus.cc:2229] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:27:57.987408 28104 raft_consensus.cc:2258] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:57.988513 28102 raft_consensus.cc:2229] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:27:57.988917 28102 raft_consensus.cc:2258] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:27:57.989719 28104 ts_tablet_manager.cc:1905] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:57.990417 28102 ts_tablet_manager.cc:1905] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:27:57.992465 28104 ts_tablet_manager.cc:1918] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:57.992583 28104 log.cc:1196] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/86e916e739ab4d9782dbb1ba42d68ef7
I20260618 22:27:57.992890 28104 ts_tablet_manager.cc:1939] T 86e916e739ab4d9782dbb1ba42d68ef7 P df8059965fbf44d7b74f5b21996f305d: Deleting consensus metadata
I20260618 22:27:57.993253 28102 ts_tablet_manager.cc:1918] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:27:57.993409 28102 log.cc:1196] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/86e916e739ab4d9782dbb1ba42d68ef7
I20260618 22:27:57.993768 28102 ts_tablet_manager.cc:1939] T 86e916e739ab4d9782dbb1ba42d68ef7 P 8b804046ed3d477896edb928a5dd2ab8: Deleting consensus metadata
W20260618 22:27:57.993810 27587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:45735 (error 108)
I20260618 22:27:57.994185 27590 catalog_manager.cc:4841] TS df8059965fbf44d7b74f5b21996f305d (127.16.243.130:41791): tablet 86e916e739ab4d9782dbb1ba42d68ef7 (table test-table [id=8b7af1b43d3f48349220f3485476b14e]) successfully deleted
I20260618 22:27:57.994675 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27799
I20260618 22:27:57.996948 27590 catalog_manager.cc:4841] TS 8b804046ed3d477896edb928a5dd2ab8 (127.16.243.131:44537): tablet 86e916e739ab4d9782dbb1ba42d68ef7 (table test-table [id=8b7af1b43d3f48349220f3485476b14e]) successfully deleted
I20260618 22:27:57.997637 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:27:57.997332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:45735 (local address 127.0.0.1:51202)
0618 22:27:57.997468 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:27:57.997545 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:45735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260618 22:27:57.997839 27587 catalog_manager.cc:4584] TS 8cfc2b25b03248f1827f2b2e21f8175a (127.16.243.129:45735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86e916e739ab4d9782dbb1ba42d68ef7: Network error: Client connection negotiation failed: client connection to 127.16.243.129:45735: connect: Connection refused (error 111)
I20260618 22:27:57.999835 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27933
I20260618 22:27:58.004340 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 27571
2026-06-18T22:27:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2142 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:27:58 dist-test-slave-jm5g krb5kdc[28108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:27:58 dist-test-slave-jm5g krb5kdc[28108](info): set up 2 sockets
Jun 18 22:27:58 dist-test-slave-jm5g krb5kdc[28108](info): commencing operation
krb5kdc: starting...
W20260618 22:27:58.963204 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:27:59 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:27:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangeradmin_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangeradmin_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangerlookup_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangerlookup_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
I20260618 22:27:59.119443 17358 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9_akNk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/postgres -l logfile start
2026-06-18 22:28:01.460 UTC [28138] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:28:01.460 UTC [28138] LOG: listening on IPv4 address "127.16.243.148", port 53749
2026-06-18 22:28:01.464 UTC [28138] LOG: listening on Unix socket "/tmp/.s.PGSQL.53749"
2026-06-18 22:28:01.483 UTC [28141] LOG: database system was shut down at 2026-06-18 22:28:00 UTC
2026-06-18 22:28:01.486 UTC [28138] LOG: database system is ready to accept connections
I20260618 22:28:02.355091 17358 mini_postgres.cc:96] Postgres bound to 53749
2026-06-18 22:28:02.360 UTC [28149] FATAL: database "slave" does not exist
127.16.243.148:53749 - accepting connections
I20260618 22:28:02.361573 17358 mini_ranger.cc:161] Starting Ranger...
I20260618 22:28:02.375828 17358 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:28:02.786949 17358 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:28:02.787029 17358 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin
2026-06-18 22:28:03,052 [I] DB FLAVOR :POSTGRES
2026-06-18 22:28:03,052 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:28:03,052 [I] Checking connection..
2026-06-18 22:28:03,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:28:03,315 [I] Checking connection passed.
2026-06-18 22:28:03,315 [I] --------- Verifying version history table ---------
2026-06-18 22:28:03,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:03,545 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:28:03,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:03,803 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:28:03,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:28:03,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:28:04.041 UTC [28213] WARNING: there is no transaction in progress
2026-06-18 22:28:04,047 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:28:04,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:04,304 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:28:04,305 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:28:04,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:28:04,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:28:04,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jm5g.c.gcp-upstream.internal','N') ;"
2026-06-18 22:28:05,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:28:05,261 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:28:05,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:28:05,520 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:28:05,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:28:05,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:28:05,998 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:28:05,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:28:06.954 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:06.970 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:06.985 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.000 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.015 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.035 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.092 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.098 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.111 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.121 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.131 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.144 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.153 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.160 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.166 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07.173 UTC [28321] WARNING: there is no transaction in progress
2026-06-18 22:28:07,824 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:28:07,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:28:08,058 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:28:08,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:28:08,299 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:28:08,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:28:08,549 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:28:08,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:28:08,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:28:09,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jm5g.c.gcp-upstream.internal';"
2026-06-18 22:28:09,291 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:28:09,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:28:09,518 [I] DB_PATCHES have already been applied
I20260618 22:28:09.523916 17358 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9_akNk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:28:09.523983 17358 mini_ranger.cc:193] Using host: 127.16.243.148
I20260618 22:28:09.527208 17358 mini_ranger.cc:227] Ranger admin URL: http://127.16.243.148:38281
Jun 18, 2026 10:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:28:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:28:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:28:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:28:10.536414 17358 mini_ranger.cc:160] Time spent starting Ranger: real 8.175s user 0.000s sys 0.009s
Jun 18, 2026 10:28:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:28:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 10:28:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:28:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:28:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:28:31.159045 17358 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangerkms_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/rangerkms_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:28:31 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:28:31.244705 17358 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:28:31.253342 17358 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:28:32.011251 17358 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:28:32.011353 17358 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms
I20260618 22:28:32.011451 17358 mini_ranger_kms.cc:227] Using postgres at 127.16.243.148:53749
2026-06-18 22:28:32,180 [I] DB FLAVOR :POSTGRES
2026-06-18 22:28:32,180 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:28:32,180 [I] Checking connection
2026-06-18 22:28:32,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:28:32,441 [I] connection success
2026-06-18 22:28:32,441 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:28:32,441 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:28:32,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:28:32,684 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:28:32,684 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:28:32,684 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:28:32,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:28:32,943 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:28:33.433543 17358 mini_ranger_kms.cc:310] Created kms service
I20260618 22:28:33.628429 17358 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:28:33.670272 17358 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:28:33 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:34.068966 17358 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:28:34.069115 17358 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms:/tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:28:34.069139 17358 mini_ranger_kms.cc:240] Using host: 127.16.243.148
I20260618 22:28:34.072607 17358 mini_ranger_kms.cc:276] Ranger KMS PID: 28501
I20260618 22:28:34.072752 17358 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.243.148:44123
Jun 18, 2026 10:28:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:28:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:28:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:28:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:28:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:28:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 18 22:28:36 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:28:36.101476 17358 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.857s user 0.025s sys 0.128s
I20260618 22:28:36.101619 17358 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:28:36.101753 17358 mini_ranger_kms.cc:401] 127.16.243.148:44123/kms/v1/keys
Jun 18 22:28:36 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
Jun 18 22:28:36 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
Jun 18 22:28:39 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:28:39.202266 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41687
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41687
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:44123/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:28:39.333412 28552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:28:39.333699 28552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:28:39.333738 28552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:28:39.337929 28552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:28:39.338093 28552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:28:39.338169 28552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:28:39.338207 28552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:28:39.338241 28552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:28:39.344313 28552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.243.148:44123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41687
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:28:39.345427 28552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:28:39.346457 28552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:28:39.355028 28552 server_base.cc:1034] running on GCE node
W20260618 22:28:39.355068 28558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:39.355067 28557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:39.355067 28560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:28:39.355870 28552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:28:39.356976 28552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:28:39.358129 28552 hybrid_clock.cc:648] HybridClock initialized: now 1781821719358107 us; error 39 us; skew 500 ppm
Jun 18 22:28:39 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:28:39.360730 28552 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:28:39.361601 28552 webserver.cc:460] Webserver started at http://127.16.243.190:42245/ using document root <none> and password file <none>
I20260618 22:28:39.362113 28552 fs_manager.cc:359] Metadata directory not provided
I20260618 22:28:39.362169 28552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:28:39.362354 28552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:28:39 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:39.419116 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "4caa367a97ee43ac81ada3dfcf5c3b44"
format_stamp: "Formatted at 2026-06-18 22:28:39 on dist-test-slave-jm5g"
server_key: "684c9a7cc8d69d45481da877ff8c9ab9"
server_key_iv: "2186e3baaba4dc07f5a6c979f21ac57e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:39.419900 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "4caa367a97ee43ac81ada3dfcf5c3b44"
format_stamp: "Formatted at 2026-06-18 22:28:39 on dist-test-slave-jm5g"
server_key: "684c9a7cc8d69d45481da877ff8c9ab9"
server_key_iv: "2186e3baaba4dc07f5a6c979f21ac57e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:39.424995 28552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:28:39.443763 28567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:39.445536 28552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:28:39.445683 28552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "4caa367a97ee43ac81ada3dfcf5c3b44"
format_stamp: "Formatted at 2026-06-18 22:28:39 on dist-test-slave-jm5g"
server_key: "684c9a7cc8d69d45481da877ff8c9ab9"
server_key_iv: "2186e3baaba4dc07f5a6c979f21ac57e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:39.445834 28552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:28:39.480913 28552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:28:39.483635 28552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:28:39.483803 28552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:28:39.490793 28552 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41687
I20260618 22:28:39.490890 28619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41687 every 8 connection(s)
I20260618 22:28:39.491739 28552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:28:39.494107 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 28552
I20260618 22:28:39.494234 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:28:39.494521 17358 external_mini_cluster.cc:1428] Setting key 4266b056e2fcb76f6237825dd5a6b093
I20260618 22:28:39.494634 28620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:28:39.500744 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: Bootstrap starting.
I20260618 22:28:39.503094 28620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: Neither blocks nor log segments found. Creating new log.
Jun 18 22:28:39 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:28:39.503813 28620 log.cc:824] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: Log is configured to *not* fsync() on all Append() calls
I20260618 22:28:39.505752 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: No bootstrap required, opened a new log
I20260618 22:28:39.508562 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:39.495976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.16.243.190:41687)
0618 22:28:39.496411 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:39.496426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:39.496472 (+ 46us) server_negotiation.cc:407] Connection header received
0618 22:28:39.497288 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:39.497307 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:39.497636 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:39.497942 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:28:39.498734 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:39.499715 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:39.500530 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:39.501022 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:39.501165 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:39.503856 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:28:39.503861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:28:39.503876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:28:39.503910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:28:39.506012 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:39.506411 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:39.506417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:39.506421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:39.506541 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:39.506806 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:39.506809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:39.506811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:39.507180 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:28:39.507368 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:39.507629 (+ 261us) server_negotiation.cc:299] Negotiation successful
0618 22:28:39.507878 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":148,"threads_started":1}
I20260618 22:28:39.508693 28620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } }
I20260618 22:28:39.508952 28620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:28:39.509018 28620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa367a97ee43ac81ada3dfcf5c3b44, State: Initialized, Role: FOLLOWER
I20260618 22:28:39.509459 28620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } }
I20260618 22:28:39.509604 28620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:28:39.509677 28620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:28:39.509773 28620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:28:39.510639 28620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } }
I20260618 22:28:39.510979 28620 leader_election.cc:302] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4caa367a97ee43ac81ada3dfcf5c3b44; no voters:
I20260618 22:28:39.511242 28620 leader_election.cc:288] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:28:39.511338 28625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:28:39.511554 28625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [term 1 LEADER]: Becoming Leader. State: Replica: 4caa367a97ee43ac81ada3dfcf5c3b44, State: Running, Role: LEADER
I20260618 22:28:39.511842 28625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } }
I20260618 22:28:39.512291 28620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:28:39.513319 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } } }
I20260618 22:28:39.513437 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [sys.catalog]: This master's current role is: LEADER
I20260618 22:28:39.513796 28634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:28:39.513764 28627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4caa367a97ee43ac81ada3dfcf5c3b44. Latest consensus state: current_term: 1 leader_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4caa367a97ee43ac81ada3dfcf5c3b44" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41687 } } }
I20260618 22:28:39.514300 28627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44 [sys.catalog]: This master's current role is: LEADER
I20260618 22:28:39.516871 28634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:28:39.517289 28620 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:28:39.526221 28634 catalog_manager.cc:1347] Generated new cluster ID: 4c07bcd25b9846f1af2248cd5f58bd91
I20260618 22:28:39.526319 28634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:28:39.536654 28634 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:28:39.537601 28634 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:28:39.544628 28634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: Generated new TSK 0
I20260618 22:28:39.545315 28634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:28:40.285421 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:44123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:28:40.435226 28665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:28:40.435485 28665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:28:40.435545 28665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:28:40.439615 28665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:28:40.439673 28665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:28:40.439754 28665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:28:40.445886 28665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.243.148:44123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:28:40.447180 28665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:28:40.448166 28665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:28:40.458487 28675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:28:40.458603 28665 server_base.cc:1034] running on GCE node
W20260618 22:28:40.458932 28673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:40.459105 28672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:28:40.459540 28665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:28:40.460203 28665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:28:40.461364 28665 hybrid_clock.cc:648] HybridClock initialized: now 1781821720461340 us; error 38 us; skew 500 ppm
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:28:40.464092 28665 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:28:40.465036 28665 webserver.cc:460] Webserver started at http://127.16.243.129:42297/ using document root <none> and password file <none>
I20260618 22:28:40.465569 28665 fs_manager.cc:359] Metadata directory not provided
I20260618 22:28:40.465633 28665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:28:40.465843 28665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:40.479635 28665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "b11e85ca5dca458bb3b203891b252528"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "aff78c0d36660a837e060dd4a80ca754"
server_key_iv: "86cbe44a51b6d8a9b1fbafcd982c84ff"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.480342 28665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "b11e85ca5dca458bb3b203891b252528"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "aff78c0d36660a837e060dd4a80ca754"
server_key_iv: "86cbe44a51b6d8a9b1fbafcd982c84ff"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.488109 28665 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260618 22:28:40.522130 28682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.523960 28665 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260618 22:28:40.524192 28665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "b11e85ca5dca458bb3b203891b252528"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "aff78c0d36660a837e060dd4a80ca754"
server_key_iv: "86cbe44a51b6d8a9b1fbafcd982c84ff"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.524420 28665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:28:40.540292 28665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:28:40.544076 28665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:28:40.544268 28665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:28:40.544845 28665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:28:40.545722 28665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:28:40.545770 28665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.545823 28665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:28:40.545846 28665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.556527 28665 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:43297
I20260618 22:28:40.557819 28665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:28:40.560539 28797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:43297 every 8 connection(s)
I20260618 22:28:40.564646 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 28665
I20260618 22:28:40.564769 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:28:40.565075 17358 external_mini_cluster.cc:1428] Setting key 85dda6271c4c20a9542c27fe82268d7e
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:28:40.582075 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:40.562953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41687 (local address 127.16.243.129:56795)
0618 22:28:40.564674 (+ 1721us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:40.564851 (+ 177us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:40.566930 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:40.567824 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:40.567835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:40.568194 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:28:40.568854 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:40.568873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.570015 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.570018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:40.571555 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:40.571566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.572324 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.572328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:40.572523 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:40.574166 (+ 1643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:28:40.574186 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:28:40.575905 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:28:40.578630 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:40.578636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:40.578657 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:40.579017 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:40.579324 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:40.579327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:40.579329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:40.579507 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:40.579991 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:28:40.579999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:28:40.580367 (+ 368us) client_negotiation.cc:769] Sending connection context
0618 22:28:40.580628 (+ 261us) client_negotiation.cc:240] Negotiation successful
0618 22:28:40.580988 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:28:40.582875 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:40.564119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:56795 (local address 127.16.243.190:41687)
0618 22:28:40.567350 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:40.567355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:40.567368 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:28:40.567405 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:40.567408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:40.567451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:40.567524 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:28:40.569031 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.569867 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.571683 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.572056 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.572150 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:40.576113 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:28:40.576117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:28:40.576120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:28:40.576147 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:28:40.578470 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:40.579134 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:40.579137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:40.579140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:40.579228 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:40.579612 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:40.579615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:40.579617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:40.579840 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:28:40.579927 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:40.582536 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0618 22:28:40.582703 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3151,"thread_start_us":83,"threads_started":1}
I20260618 22:28:40.584797 28798 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41687
I20260618 22:28:40.585074 28798 heartbeater.cc:464] Registering TS with master...
I20260618 22:28:40.585666 28798 heartbeater.cc:511] Master 127.16.243.190:41687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:28:40.588994 28584 ts_manager.cc:194] Registered new tserver with Master: b11e85ca5dca458bb3b203891b252528 (127.16.243.129:43297)
I20260618 22:28:40.590540 28584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:56795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:28:40.604308 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:44123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 22:28:40.757872 28804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:28:40.758237 28804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:28:40.758301 28804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:28:40.764447 28804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:28:40.764598 28804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:28:40.764714 28804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:28:40.772189 28804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.243.148:44123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:28:40.773595 28804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:28:40.774937 28804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:28:40.787111 28811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:40.787268 28810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:40.787446 28813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:28:40.788159 28804 server_base.cc:1034] running on GCE node
I20260618 22:28:40.788681 28804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:28:40.789516 28804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:28:40.792794 28804 hybrid_clock.cc:648] HybridClock initialized: now 1781821720792709 us; error 96 us; skew 500 ppm
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:28:40.796836 28804 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:28:40.798128 28804 webserver.cc:460] Webserver started at http://127.16.243.130:33897/ using document root <none> and password file <none>
I20260618 22:28:40.798945 28804 fs_manager.cc:359] Metadata directory not provided
I20260618 22:28:40.799078 28804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:28:40.799348 28804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:40.820690 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "c958662b788b40eeb03e1206c56baaf8"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "2631dcbffad69fefeffc7c631638bab7"
server_key_iv: "f8a37edb21108b4e94a0b81cac0a1b1e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.821492 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "c958662b788b40eeb03e1206c56baaf8"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "2631dcbffad69fefeffc7c631638bab7"
server_key_iv: "f8a37edb21108b4e94a0b81cac0a1b1e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.826926 28804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:28:40.845839 28820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.847679 28804 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:28:40.847843 28804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "c958662b788b40eeb03e1206c56baaf8"
format_stamp: "Formatted at 2026-06-18 22:28:40 on dist-test-slave-jm5g"
server_key: "2631dcbffad69fefeffc7c631638bab7"
server_key_iv: "f8a37edb21108b4e94a0b81cac0a1b1e"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:40.848001 28804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:28:40.873831 28804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:28:40.877715 28804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:28:40.878012 28804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:28:40.878778 28804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:28:40.879937 28804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:28:40.880005 28804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.880065 28804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:28:40.880101 28804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:40.891973 28804 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45649
I20260618 22:28:40.892017 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45649 every 8 connection(s)
I20260618 22:28:40.893020 28804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:28:40.894398 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 28804
I20260618 22:28:40.894519 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:28:40.894840 17358 external_mini_cluster.cc:1428] Setting key 0c1bf695d0fcb5c5c5d656493c12909d
Jun 18 22:28:40 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:28:40.906859 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:40.894978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:58535 (local address 127.16.243.190:41687)
0618 22:28:40.895120 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:40.895125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:40.896147 (+ 1022us) server_negotiation.cc:407] Connection header received
0618 22:28:40.897140 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:40.897144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:40.897199 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:40.897278 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:28:40.898583 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.899448 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.900654 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.901008 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.901125 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:40.903464 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:28:40.903467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:28:40.903469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:28:40.903493 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:28:40.904870 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:40.905347 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:40.905351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:40.905353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:40.905441 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:40.905754 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:40.905757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:40.905759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:40.905942 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:28:40.906025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:40.906526 (+ 501us) server_negotiation.cc:299] Negotiation successful
0618 22:28:40.906691 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:28:40.907814 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:40.895482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41687 (local address 127.16.243.130:58535)
0618 22:28:40.895986 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:40.896044 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:40.896929 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:40.897469 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:40.897477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:40.897759 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:28:40.898359 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:40.898393 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.899604 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.899607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:40.900524 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:40.900534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:40.901175 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:40.901178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:40.901328 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:40.901935 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:28:40.901953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:28:40.903292 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:28:40.904983 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:40.904989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:40.905007 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:40.905253 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:40.905522 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:40.905525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:40.905527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:40.905670 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:40.906037 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:28:40.906044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:28:40.906290 (+ 246us) client_negotiation.cc:769] Sending connection context
0618 22:28:40.906461 (+ 171us) client_negotiation.cc:240] Negotiation successful
0618 22:28:40.906807 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260618 22:28:40.908954 28934 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41687
I20260618 22:28:40.909189 28934 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:28:40.909699 28934 heartbeater.cc:511] Master 127.16.243.190:41687 requested a full tablet report, sending...
I20260618 22:28:40.910805 28584 ts_manager.cc:194] Registered new tserver with Master: c958662b788b40eeb03e1206c56baaf8 (127.16.243.130:45649)
I20260618 22:28:40.911582 28584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:58535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:28:40.938498 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:44123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:28:41 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
W20260618 22:28:41.108543 28939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:28:41.108933 28939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:28:41.109006 28939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:28:41.115127 28939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:28:41.115214 28939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:28:41.115300 28939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:28:41.123243 28939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.243.148:44123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:28:41.124893 28939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:28:41.126375 28939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:28:41.140337 28947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:41.140838 28945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:28:41.140352 28944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:28:41.141010 28939 server_base.cc:1034] running on GCE node
I20260618 22:28:41.141553 28939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:28:41.142377 28939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:28:41.143620 28939 hybrid_clock.cc:648] HybridClock initialized: now 1781821721143584 us; error 51 us; skew 500 ppm
Jun 18 22:28:41 dist-test-slave-jm5g krb5kdc[28108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:28:41.147310 28939 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:28:41.148481 28939 webserver.cc:460] Webserver started at http://127.16.243.131:36521/ using document root <none> and password file <none>
I20260618 22:28:41.149166 28939 fs_manager.cc:359] Metadata directory not provided
I20260618 22:28:41.149243 28939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:28:41.149518 28939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:28:41 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:41.167243 28939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "55b10b74d1d04c75bc62728d3b7a49b4"
format_stamp: "Formatted at 2026-06-18 22:28:41 on dist-test-slave-jm5g"
server_key: "05ee97f4d413641dfe422d15b59a3ad0"
server_key_iv: "1cd425701fbfe96ae4f564a349e333bd"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:41.168088 28939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "55b10b74d1d04c75bc62728d3b7a49b4"
format_stamp: "Formatted at 2026-06-18 22:28:41 on dist-test-slave-jm5g"
server_key: "05ee97f4d413641dfe422d15b59a3ad0"
server_key_iv: "1cd425701fbfe96ae4f564a349e333bd"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:41.173313 28939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:28:41.199079 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:41.200850 28939 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260618 22:28:41.200997 28939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "55b10b74d1d04c75bc62728d3b7a49b4"
format_stamp: "Formatted at 2026-06-18 22:28:41 on dist-test-slave-jm5g"
server_key: "05ee97f4d413641dfe422d15b59a3ad0"
server_key_iv: "1cd425701fbfe96ae4f564a349e333bd"
server_key_version: "kuduclusterkey@0"
I20260618 22:28:41.201135 28939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:28:41.228379 28939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:28:41.232098 28939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:28:41.232300 28939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:28:41.232933 28939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:28:41.233929 28939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:28:41.233994 28939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:41.234045 28939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:28:41.234115 28939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:28:41.246524 28939 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:41781
I20260618 22:28:41.246587 29067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:41781 every 8 connection(s)
I20260618 22:28:41.247772 28939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:28:41.249469 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 28939
I20260618 22:28:41.249610 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:28:41.249908 17358 external_mini_cluster.cc:1428] Setting key 2fc4bddefe394e37d468073f9fb010fa
Jun 18 22:28:41 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821721, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:28:41.268566 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.249767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50667 (local address 127.16.243.190:41687)
0618 22:28:41.249909 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.249913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.252741 (+ 2828us) server_negotiation.cc:407] Connection header received
0618 22:28:41.254180 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.254183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.254239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.254291 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:28:41.257594 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.258413 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.260227 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.260585 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.260670 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.264078 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:28:41.264083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:28:41.264086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:28:41.264116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:28:41.265969 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:41.266657 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:41.266662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:41.266663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:41.266765 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:41.267221 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:41.267225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:41.267227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:41.267485 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:28:41.267590 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.268245 (+ 655us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.268386 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:28:41.269553 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.250310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41687 (local address 127.16.243.131:50667)
0618 22:28:41.252563 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.252625 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.253930 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.254783 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.254793 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.255149 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:28:41.255863 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.255885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.258626 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.258629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.260081 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.260092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.260901 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.260905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.261116 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.261816 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:28:41.261838 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:28:41.263646 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:28:41.266070 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:41.266077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:41.266099 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:41.266474 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:41.266866 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:28:41.266869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:28:41.266871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:28:41.267096 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:28:41.267585 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:28:41.267593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:28:41.267936 (+ 343us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.268147 (+ 211us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.268494 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"thread_start_us":885,"threads_started":1}
I20260618 22:28:41.270859 29068 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41687
I20260618 22:28:41.271133 29068 heartbeater.cc:464] Registering TS with master...
I20260618 22:28:41.271708 29068 heartbeater.cc:511] Master 127.16.243.190:41687 requested a full tablet report, sending...
I20260618 22:28:41.272954 28584 ts_manager.cc:194] Registered new tserver with Master: 55b10b74d1d04c75bc62728d3b7a49b4 (127.16.243.131:41781)
I20260618 22:28:41.273686 28584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:50667
I20260618 22:28:41.275488 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:28:41.287345 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.277747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.16.243.190:41687)
0618 22:28:41.277855 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.277859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.278127 (+ 268us) server_negotiation.cc:407] Connection header received
0618 22:28:41.278230 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.278233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.278283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.278362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:28:41.279208 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.280016 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.281151 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.281476 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.281580 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.283449 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:28:41.283453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:28:41.283456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:28:41.283484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:28:41.285332 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:41.285854 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:41.285858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:41.285860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:41.285958 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:28:41.286242 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:28:41.286246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:28:41.286248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:28:41.286491 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:28:41.286637 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.287055 (+ 418us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.287180 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:28:41.291126 28584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:28:41.295722 28584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:28:41.317371 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.307211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43297 (local address 127.0.0.1:40434)
0618 22:28:41.311054 (+ 3843us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.311115 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.311255 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.312537 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.312545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.312571 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:28:41.312724 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.312731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.314076 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.314080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.315630 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.315638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.316853 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.316857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.316957 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.316992 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.317092 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.317212 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3635,"thread_start_us":113,"threads_started":1}
I20260618 22:28:41.317838 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.307346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.16.243.129:43297)
0618 22:28:41.311698 (+ 4352us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.311706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.311737 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:28:41.311828 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.311834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.312133 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.312307 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:28:41.312861 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.313956 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.315750 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.316556 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.316670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.316798 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.317594 (+ 796us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.317703 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4218,"thread_start_us":150,"threads_started":1}
I20260618 22:28:41.318358 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.308771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45649 (local address 127.0.0.1:49478)
0618 22:28:41.311686 (+ 2915us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.311698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.311782 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.313119 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.313122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.313140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:28:41.313225 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.313231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.314953 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.314957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.316693 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.316703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.318000 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.318004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.318084 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.318108 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.318175 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.318237 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2860,"thread_start_us":90,"threads_started":1}
I20260618 22:28:41.318953 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.309174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.16.243.130:45649)
0618 22:28:41.311556 (+ 2382us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.311567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.311867 (+ 300us) server_negotiation.cc:407] Connection header received
0618 22:28:41.311948 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.311953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.312162 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.312985 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:28:41.313362 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.314812 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.316866 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.317706 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.317811 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.317939 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.318695 (+ 756us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.318798 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2274,"thread_start_us":129,"threads_started":1}
I20260618 22:28:41.319697 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.309990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:41781 (local address 127.0.0.1:40328)
0618 22:28:41.312556 (+ 2566us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.312569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.312645 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.314114 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.314117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.314133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:28:41.314206 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.314211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.316041 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.316045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.317088 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.317096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.319431 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.319434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.319499 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.319515 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.319560 (+ 45us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.319610 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2507,"thread_start_us":78,"threads_started":1}
I20260618 22:28:41.320124 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.310128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.16.243.131:41781)
0618 22:28:41.313529 (+ 3401us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.313541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.313563 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:28:41.313634 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.313639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.313833 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.313980 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:28:41.314319 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.315466 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.317235 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.319160 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.319271 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.319875 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.319935 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.320017 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3290,"thread_start_us":92,"threads_started":1}
I20260618 22:28:41.321452 28732 tablet_service.cc:1467] Processing CreateTablet for tablet fe4741e38983458896d659919d30acf5 (DEFAULT_TABLE table=test-table [id=936c9d3492424eaa839928b01fe7b756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:28:41.322703 28732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4741e38983458896d659919d30acf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:28:41.323004 29002 tablet_service.cc:1467] Processing CreateTablet for tablet fe4741e38983458896d659919d30acf5 (DEFAULT_TABLE table=test-table [id=936c9d3492424eaa839928b01fe7b756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:28:41.324031 29002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4741e38983458896d659919d30acf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:28:41.336375 29087 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Bootstrap starting.
I20260618 22:28:41.338186 28868 tablet_service.cc:1467] Processing CreateTablet for tablet fe4741e38983458896d659919d30acf5 (DEFAULT_TABLE table=test-table [id=936c9d3492424eaa839928b01fe7b756]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:28:41.339188 29087 tablet_bootstrap.cc:654] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Neither blocks nor log segments found. Creating new log.
I20260618 22:28:41.339376 28868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4741e38983458896d659919d30acf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:28:41.340047 29087 log.cc:824] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:28:41.342396 29087 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: No bootstrap required, opened a new log
I20260618 22:28:41.342598 29087 ts_tablet_manager.cc:1397] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260618 22:28:41.346688 29087 raft_consensus.cc:348] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.348146 29087 raft_consensus.cc:374] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:28:41.348241 29087 raft_consensus.cc:729] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b10b74d1d04c75bc62728d3b7a49b4, State: Initialized, Role: FOLLOWER
I20260618 22:28:41.348856 29087 consensus_queue.cc:260] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.350091 29087 ts_tablet_manager.cc:1428] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260618 22:28:41.350492 29068 heartbeater.cc:503] Master 127.16.243.190:41687 was elected leader, sending a full tablet report...
I20260618 22:28:41.353891 29088 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: Bootstrap starting.
I20260618 22:28:41.354301 29091 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Bootstrap starting.
I20260618 22:28:41.358170 29091 tablet_bootstrap.cc:654] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Neither blocks nor log segments found. Creating new log.
I20260618 22:28:41.359169 29091 log.cc:824] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:28:41.361409 29088 tablet_bootstrap.cc:654] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: Neither blocks nor log segments found. Creating new log.
I20260618 22:28:41.362286 29088 log.cc:824] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: Log is configured to *not* fsync() on all Append() calls
I20260618 22:28:41.365762 29091 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: No bootstrap required, opened a new log
I20260618 22:28:41.366050 29091 ts_tablet_manager.cc:1397] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
Jun 18 22:28:41 dist-test-slave-jm5g krb5kdc[28108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:28:41.368284 29088 tablet_bootstrap.cc:492] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: No bootstrap required, opened a new log
I20260618 22:28:41.368496 29088 ts_tablet_manager.cc:1397] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260618 22:28:41.373077 29088 raft_consensus.cc:348] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.373574 29088 raft_consensus.cc:374] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:28:41.373677 29088 raft_consensus.cc:729] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11e85ca5dca458bb3b203891b252528, State: Initialized, Role: FOLLOWER
I20260618 22:28:41.374289 29088 consensus_queue.cc:260] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.374506 29091 raft_consensus.cc:348] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.374864 29091 raft_consensus.cc:374] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:28:41.374963 29091 raft_consensus.cc:729] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c958662b788b40eeb03e1206c56baaf8, State: Initialized, Role: FOLLOWER
I20260618 22:28:41.375516 29091 consensus_queue.cc:260] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.375701 29088 ts_tablet_manager.cc:1428] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260618 22:28:41.376528 29091 ts_tablet_manager.cc:1428] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260618 22:28:41.376576 28934 heartbeater.cc:503] Master 127.16.243.190:41687 was elected leader, sending a full tablet report...
I20260618 22:28:41.377344 28798 heartbeater.cc:503] Master 127.16.243.190:41687 was elected leader, sending a full tablet report...
W20260618 22:28:41.394236 28935 tablet.cc:2369] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:28:41.405467 29094 raft_consensus.cc:482] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:28:41.405720 29094 raft_consensus.cc:504] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.407191 29094 leader_election.cc:288] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c958662b788b40eeb03e1206c56baaf8 (127.16.243.130:45649), 55b10b74d1d04c75bc62728d3b7a49b4 (127.16.243.131:41781)
I20260618 22:28:41.413600 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.407603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:56489 (local address 127.16.243.131:41781)
0618 22:28:41.407725 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.407730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.408933 (+ 1203us) server_negotiation.cc:407] Connection header received
0618 22:28:41.409133 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.409137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.409187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.409231 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:28:41.409638 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.410456 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.412152 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.412761 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.412861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.412929 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.413374 (+ 445us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.413462 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:28:41.414383 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.407550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:41781 (local address 127.16.243.129:56489)
0618 22:28:41.408837 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.408851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.409016 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.409361 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.409364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.409380 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:28:41.409517 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.409523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.410638 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.410642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.412005 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.412016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.413081 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.413085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.413184 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.413207 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.414024 (+ 817us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.414189 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":49,"thread_start_us":139,"threads_started":1}
I20260618 22:28:41.415647 29012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4741e38983458896d659919d30acf5" candidate_uuid: "b11e85ca5dca458bb3b203891b252528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" is_pre_election: true
I20260618 22:28:41.416015 29012 raft_consensus.cc:2454] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b11e85ca5dca458bb3b203891b252528 in term 0.
I20260618 22:28:41.416441 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.408417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:44701 (local address 127.16.243.130:45649)
0618 22:28:41.408546 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.408550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.412331 (+ 3781us) server_negotiation.cc:407] Connection header received
0618 22:28:41.412574 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.412578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.412631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.412678 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:28:41.413052 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.413776 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.415265 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.415800 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.415890 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.415936 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.416252 (+ 316us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.416330 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:28:41.416666 28684 leader_election.cc:302] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55b10b74d1d04c75bc62728d3b7a49b4, b11e85ca5dca458bb3b203891b252528; no voters:
I20260618 22:28:41.416837 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.408059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45649 (local address 127.16.243.129:44701)
0618 22:28:41.412236 (+ 4177us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.412252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:28:41.412423 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:28:41.412807 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:28:41.412810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:28:41.412826 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:28:41.412934 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.412939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.413912 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.413916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.415129 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:28:41.415140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.416025 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.416028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:28:41.416121 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.416142 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:28:41.416623 (+ 481us) client_negotiation.cc:240] Negotiation successful
0618 22:28:41.416696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4099,"mutex_wait_us":35,"thread_start_us":97,"threads_started":1}
I20260618 22:28:41.416985 29094 raft_consensus.cc:2781] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:28:41.417064 29094 raft_consensus.cc:482] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:28:41.417105 29094 raft_consensus.cc:3037] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:28:41.417369 28888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4741e38983458896d659919d30acf5" candidate_uuid: "b11e85ca5dca458bb3b203891b252528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c958662b788b40eeb03e1206c56baaf8" is_pre_election: true
I20260618 22:28:41.417642 28888 raft_consensus.cc:2454] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b11e85ca5dca458bb3b203891b252528 in term 0.
I20260618 22:28:41.418388 29094 raft_consensus.cc:504] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.418745 29094 leader_election.cc:288] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [CANDIDATE]: Term 1 election: Requested vote from peers c958662b788b40eeb03e1206c56baaf8 (127.16.243.130:45649), 55b10b74d1d04c75bc62728d3b7a49b4 (127.16.243.131:41781)
I20260618 22:28:41.419329 28888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4741e38983458896d659919d30acf5" candidate_uuid: "b11e85ca5dca458bb3b203891b252528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c958662b788b40eeb03e1206c56baaf8"
I20260618 22:28:41.419442 28888 raft_consensus.cc:3037] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:28:41.420573 28888 raft_consensus.cc:2454] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b11e85ca5dca458bb3b203891b252528 in term 1.
I20260618 22:28:41.420943 28686 leader_election.cc:302] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b11e85ca5dca458bb3b203891b252528, c958662b788b40eeb03e1206c56baaf8; no voters:
I20260618 22:28:41.421473 29012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4741e38983458896d659919d30acf5" candidate_uuid: "b11e85ca5dca458bb3b203891b252528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b10b74d1d04c75bc62728d3b7a49b4"
I20260618 22:28:41.421593 29012 raft_consensus.cc:3037] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:28:41.422959 29012 raft_consensus.cc:2454] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b11e85ca5dca458bb3b203891b252528 in term 1.
I20260618 22:28:41.423538 29094 raft_consensus.cc:2781] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:28:41.423863 29094 raft_consensus.cc:686] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [term 1 LEADER]: Becoming Leader. State: Replica: b11e85ca5dca458bb3b203891b252528, State: Running, Role: LEADER
I20260618 22:28:41.424335 29094 consensus_queue.cc:237] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } }
I20260618 22:28:41.427909 28583 catalog_manager.cc:5495] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 reported cstate change: term changed from 0 to 1, leader changed from <none> to b11e85ca5dca458bb3b203891b252528 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "b11e85ca5dca458bb3b203891b252528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b11e85ca5dca458bb3b203891b252528" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 43297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 } health_report { overall_health: UNKNOWN } } }
I20260618 22:28:41.464970 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.459944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.16.243.129:43297)
0618 22:28:41.460073 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:28:41.460078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:28:41.460093 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:28:41.460191 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:28:41.460194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:28:41.460270 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:28:41.460381 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:28:41.460717 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.461499 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.462742 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:28:41.463071 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:28:41.463170 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:28:41.463617 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:28:41.464214 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:28:41.464326 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:28:41.464759 (+ 433us) server_negotiation.cc:299] Negotiation successful
0618 22:28:41.464822 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:28:41.487210 29012 raft_consensus.cc:1261] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 1 FOLLOWER]: Refusing update from remote peer b11e85ca5dca458bb3b203891b252528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:28:41.487321 28888 raft_consensus.cc:1261] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 1 FOLLOWER]: Refusing update from remote peer b11e85ca5dca458bb3b203891b252528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:28:41.488052 29099 consensus_queue.cc:1035] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c958662b788b40eeb03e1206c56baaf8" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:28:41.488518 29099 consensus_queue.cc:1035] T fe4741e38983458896d659919d30acf5 P b11e85ca5dca458bb3b203891b252528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b10b74d1d04c75bc62728d3b7a49b4" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:28:41.502517 29069 tablet.cc:2369] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:28:41.512377 29102 mvcc.cc:205] Tried to move back new op lower bound from 7298341771202428928 to 7298341770957983744. Current Snapshot: MvccSnapshot[applied={T|T < 7298341771202428928}]
I20260618 22:28:41.515905 29105 mvcc.cc:205] Tried to move back new op lower bound from 7298341771202428928 to 7298341770957983744. Current Snapshot: MvccSnapshot[applied={T|T < 7298341771202428928}]
I20260618 22:28:41.535807 28583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:28:41.535985 28583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:28:41.543764 28583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4caa367a97ee43ac81ada3dfcf5c3b44: Sending DeleteTablet for 3 replicas of tablet fe4741e38983458896d659919d30acf5
I20260618 22:28:41.544972 28868 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4741e38983458896d659919d30acf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:28:41 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:49478
I20260618 22:28:41.546103 29002 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4741e38983458896d659919d30acf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:28:41 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:40328
I20260618 22:28:41.547159 28732 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4741e38983458896d659919d30acf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:28:41 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:40434
I20260618 22:28:41.548828 29112 tablet_replica.cc:330] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: stopping tablet replica
I20260618 22:28:41.549016 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 28665
I20260618 22:28:41.550208 29111 tablet_replica.cc:330] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8: stopping tablet replica
I20260618 22:28:41.550542 29111 raft_consensus.cc:2229] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:28:41.550834 29112 raft_consensus.cc:2229] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:28:41.551282 29112 raft_consensus.cc:2258] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:28:41.553628 29112 ts_tablet_manager.cc:1905] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:28:41.556851 29112 ts_tablet_manager.cc:1918] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:28:41.557063 29112 log.cc:1196] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/fe4741e38983458896d659919d30acf5
I20260618 22:28:41.557497 29112 ts_tablet_manager.cc:1939] T fe4741e38983458896d659919d30acf5 P 55b10b74d1d04c75bc62728d3b7a49b4: Deleting consensus metadata
I20260618 22:28:41.558916 29111 raft_consensus.cc:2258] T fe4741e38983458896d659919d30acf5 P c958662b788b40eeb03e1206c56baaf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:28:41.559091 28569 catalog_manager.cc:4841] TS 55b10b74d1d04c75bc62728d3b7a49b4 (127.16.243.131:41781): tablet fe4741e38983458896d659919d30acf5 (table test-table [id=936c9d3492424eaa839928b01fe7b756]) successfully deleted
W20260618 22:28:41.563681 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:43297 (error 108)
I20260618 22:28:41.564018 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 28804
I20260618 22:28:41.567526 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.567170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:43297 (local address 127.0.0.1:40462)
0618 22:28:41.567360 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.567434 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260618 22:28:41.567713 28569 catalog_manager.cc:4584] TS b11e85ca5dca458bb3b203891b252528 (127.16.243.129:43297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe4741e38983458896d659919d30acf5: Network error: Client connection negotiation failed: client connection to 127.16.243.129:43297: connect: Connection refused (error 111)
I20260618 22:28:41.570511 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 28939
I20260618 22:28:41.571820 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:28:41.570713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45649 (local address 127.0.0.1:49494)
0618 22:28:41.571677 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0618 22:28:41.571734 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":882}
I20260618 22:28:41.575569 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 28552
I20260618 22:28:41.580546 17358 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:28:41.915819 17358 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:28:41.915942 17358 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:28:42.250785 17358 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:28:42.251 UTC [28138] LOG: received smart shutdown request
2026-06-18 22:28:42.254 UTC [28138] LOG: background worker "logical replication launcher" (PID 28147) exited with exit code 1
2026-06-18 22:28:42.254 UTC [28142] LOG: shutting down
2026-06-18 22:28:42.402 UTC [28138] LOG: database system is shut down
2026-06-18T22:28:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44488 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:28:42 dist-test-slave-jm5g krb5kdc[29123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:28:42 dist-test-slave-jm5g krb5kdc[29123](info): set up 2 sockets
Jun 18 22:28:42 dist-test-slave-jm5g krb5kdc[29123](info): commencing operation
krb5kdc: starting...
W20260618 22:28:43.454625 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:28:43 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:28:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangeradmin_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangeradmin_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangerlookup_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangerlookup_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
I20260618 22:28:43.610536 17358 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9_akNk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/postgres -l logfile start
2026-06-18 22:28:45.814 UTC [29153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:28:45.814 UTC [29153] LOG: listening on IPv4 address "127.16.243.148", port 38027
2026-06-18 22:28:45.817 UTC [29153] LOG: listening on Unix socket "/tmp/.s.PGSQL.38027"
2026-06-18 22:28:45.844 UTC [29156] LOG: database system was shut down at 2026-06-18 22:28:44 UTC
2026-06-18 22:28:45.847 UTC [29153] LOG: database system is ready to accept connections
I20260618 22:28:46.718119 17358 mini_postgres.cc:96] Postgres bound to 38027
2026-06-18 22:28:46.723 UTC [29164] FATAL: database "slave" does not exist
127.16.243.148:38027 - accepting connections
I20260618 22:28:46.724357 17358 mini_ranger.cc:161] Starting Ranger...
I20260618 22:28:46.738355 17358 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:28:47.127218 17358 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:28:47.127285 17358 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin
2026-06-18 22:28:47,384 [I] DB FLAVOR :POSTGRES
2026-06-18 22:28:47,384 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:28:47,384 [I] Checking connection..
2026-06-18 22:28:47,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:28:47,640 [I] Checking connection passed.
2026-06-18 22:28:47,640 [I] --------- Verifying version history table ---------
2026-06-18 22:28:47,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:47,887 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:28:47,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:48,123 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:28:48,124 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:28:48,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:28:48.352 UTC [29228] WARNING: there is no transaction in progress
2026-06-18 22:28:48,358 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:28:48,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:28:48,604 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:28:48,604 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:28:48,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:28:48,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:28:49,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jm5g.c.gcp-upstream.internal','N') ;"
2026-06-18 22:28:49,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:28:49,568 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:28:49,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:28:49,804 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:28:49,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:28:50,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:28:50,296 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:28:50,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:28:51.197 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.212 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.229 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.244 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.259 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.276 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.330 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.336 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.351 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.360 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.371 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.384 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.393 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.399 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.408 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:51.416 UTC [29336] WARNING: there is no transaction in progress
2026-06-18 22:28:52,095 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:28:52,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:28:52,328 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:28:52,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:28:52,589 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:28:52,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:28:52,830 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:28:52,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:28:53,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:28:53,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jm5g.c.gcp-upstream.internal';"
2026-06-18 22:28:53,548 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:28:53,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:28:53,797 [I] DB_PATCHES have already been applied
I20260618 22:28:53.803337 17358 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9_akNk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:28:53.803471 17358 mini_ranger.cc:193] Using host: 127.16.243.148
I20260618 22:28:53.806770 17358 mini_ranger.cc:227] Ranger admin URL: http://127.16.243.148:39965
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:28:54.799878 17358 mini_ranger.cc:160] Time spent starting Ranger: real 8.076s user 0.000s sys 0.008s
Jun 18, 2026 10:29:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:29:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 10:29:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:29:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:29:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:29:16.389396 17358 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangerkms_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/rangerkms_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/HTTP_127.16.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:29:16 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821756, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:29:16.466542 17358 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:29:16.475628 17358 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:29:17.192679 17358 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:29:17.192760 17358 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms
I20260618 22:29:17.192778 17358 mini_ranger_kms.cc:227] Using postgres at 127.16.243.148:38027
2026-06-18 22:29:17,344 [I] DB FLAVOR :POSTGRES
2026-06-18 22:29:17,344 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:29:17,344 [I] Checking connection
2026-06-18 22:29:17,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:29:17,590 [I] connection success
2026-06-18 22:29:17,590 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:29:17,590 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:29:17,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:29:17,818 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:29:17,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:29:17,818 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:29:17,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.243.148:38027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:29:18,091 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:29:18.530808 17358 mini_ranger_kms.cc:310] Created kms service
I20260618 22:29:18.771052 17358 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:29:18.813278 17358 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:29:18 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821756, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:19.223479 17358 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:29:19.223743 17358 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms:/tmp/dist-test-task9_akNk/build/debug/bin/postgresql.jar:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9_akNk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:29:19.223768 17358 mini_ranger_kms.cc:240] Using host: 127.16.243.148
I20260618 22:29:19.227172 17358 mini_ranger_kms.cc:276] Ranger KMS PID: 29516
I20260618 22:29:19.227260 17358 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.243.148:39513
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 22:29:21.229715 17358 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.763s user 0.021s sys 0.115s
I20260618 22:29:21.229851 17358 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:29:21.229923 17358 mini_ranger_kms.cc:401] 127.16.243.148:39513/kms/v1/keys
Jun 18 22:29:21 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 22:29:21 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
Jun 18 22:29:22 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.243.148@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
Jun 18 22:29:24 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:29:24.432777 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45257
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:45257
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:24.547376 29567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:24.547657 29567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:24.547695 29567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:24.551868 29567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:24.551940 29567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:24.551959 29567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:24.551973 29567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:24.551987 29567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:24.557641 29567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.243.148:39513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:45257
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:24.558806 29567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:24.559890 29567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:24.568823 29575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:24.568847 29573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:24.568948 29572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:24.569020 29567 server_base.cc:1034] running on GCE node
I20260618 22:29:24.569890 29567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:24.571321 29567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:24.572499 29567 hybrid_clock.cc:648] HybridClock initialized: now 1781821764572473 us; error 43 us; skew 500 ppm
Jun 18 22:29:24 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821764, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:24.577971 29567 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:24.578986 29567 webserver.cc:460] Webserver started at http://127.16.243.190:35553/ using document root <none> and password file <none>
I20260618 22:29:24.579541 29567 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:24.579599 29567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:24.579799 29567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:29:24 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821764, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:24.630676 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "4442a721287c4edaa996c7da6d6162b8"
format_stamp: "Formatted at 2026-06-18 22:29:24 on dist-test-slave-jm5g"
server_key: "1825218294cf9a5e23b6069cf0dae96d"
server_key_iv: "26e13601794ad670eecf861350921bb6"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:24.631279 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "4442a721287c4edaa996c7da6d6162b8"
format_stamp: "Formatted at 2026-06-18 22:29:24 on dist-test-slave-jm5g"
server_key: "1825218294cf9a5e23b6069cf0dae96d"
server_key_iv: "26e13601794ad670eecf861350921bb6"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:24.635274 29567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:29:24.649974 29582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:24.651499 29567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:29:24.651654 29567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "4442a721287c4edaa996c7da6d6162b8"
format_stamp: "Formatted at 2026-06-18 22:29:24 on dist-test-slave-jm5g"
server_key: "1825218294cf9a5e23b6069cf0dae96d"
server_key_iv: "26e13601794ad670eecf861350921bb6"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:24.651813 29567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:24.678435 29567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:24.681605 29567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:24.682089 29567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:24.689793 29567 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:45257
I20260618 22:29:24.689833 29634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:45257 every 8 connection(s)
I20260618 22:29:24.690907 29567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:24.694414 29635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:24.700614 29635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Bootstrap starting.
I20260618 22:29:24.701133 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 29567
I20260618 22:29:24.701246 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:29:24.701498 17358 external_mini_cluster.cc:1428] Setting key 320f0ba8bee5b074099c2cb6daf0c347
I20260618 22:29:24.703315 29635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:24.704238 29635 log.cc:824] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:24.706573 29635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: No bootstrap required, opened a new log
Jun 18 22:29:24 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:24.710462 29635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:24.710733 29635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:24.710824 29635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4442a721287c4edaa996c7da6d6162b8, State: Initialized, Role: FOLLOWER
I20260618 22:29:24.711289 29635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:24.711460 29635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:24.711529 29635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:24.711633 29635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:24.712741 29635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:24.713119 29635 leader_election.cc:302] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4442a721287c4edaa996c7da6d6162b8; no voters:
I20260618 22:29:24.713462 29635 leader_election.cc:288] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:29:24.713574 29640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:24.713773 29640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4442a721287c4edaa996c7da6d6162b8, State: Running, Role: LEADER
I20260618 22:29:24.714131 29640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:24.714887 29635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:24.715538 29642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4442a721287c4edaa996c7da6d6162b8. Latest consensus state: current_term: 1 leader_uuid: "4442a721287c4edaa996c7da6d6162b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } } }
I20260618 22:29:24.716284 29642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:24.716416 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:24.702874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.16.243.190:45257)
0618 22:29:24.703304 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:24.703322 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:24.703365 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:29:24.704247 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:24.704272 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:24.704724 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:24.705042 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:24.705766 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:24.707195 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:24.708072 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:24.708751 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:24.708919 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:24.711267 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:24.711274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:24.711290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:24.711322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:24.713123 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:24.713610 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:24.713616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:24.713622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:24.713772 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:24.714083 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:24.714086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:24.714087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:24.714602 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:24.714797 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:24.715121 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 22:29:24.715402 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260618 22:29:24.716180 29641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4442a721287c4edaa996c7da6d6162b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } } }
I20260618 22:29:24.716542 29641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:24.716770 29651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:24.717099 29635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:29:24.719672 29651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:24.728592 29651 catalog_manager.cc:1347] Generated new cluster ID: 403e76e2f30647d5b05f242c74ca1911
I20260618 22:29:24.728741 29651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:24.757417 29651 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:29:24.758805 29651 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:24.774022 29651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Generated new TSK 0
I20260618 22:29:24.774995 29651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:29:25.453771 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:25.595866 29680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:25.596282 29680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:25.596422 29680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:25.602339 29680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:25.602427 29680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:25.602523 29680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:25.609560 29680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:25.610921 29680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:25.611974 29680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:25.621968 29688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:25.621968 29690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:25.622498 29687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:25.623008 29680 server_base.cc:1034] running on GCE node
I20260618 22:29:25.623474 29680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:25.624179 29680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:25.625351 29680 hybrid_clock.cc:648] HybridClock initialized: now 1781821765625325 us; error 41 us; skew 500 ppm
Jun 18 22:29:25 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:25.628268 29680 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:25.629297 29680 webserver.cc:460] Webserver started at http://127.16.243.129:41079/ using document root <none> and password file <none>
I20260618 22:29:25.629851 29680 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:25.629925 29680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:25.630136 29680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:29:25 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:25.652652 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "15aacffd2f144387ad987118275684bf"
format_stamp: "Formatted at 2026-06-18 22:29:25 on dist-test-slave-jm5g"
server_key: "b167169262307d942d3f09c913c56938"
server_key_iv: "e03bdad7c7d2d92b4c963c595d8ac3ac"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:25.653672 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "15aacffd2f144387ad987118275684bf"
format_stamp: "Formatted at 2026-06-18 22:29:25 on dist-test-slave-jm5g"
server_key: "b167169262307d942d3f09c913c56938"
server_key_iv: "e03bdad7c7d2d92b4c963c595d8ac3ac"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:25.659830 29680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260618 22:29:25.695101 29697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:25.700052 29680 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.000s sys 0.004s
I20260618 22:29:25.700256 29680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "15aacffd2f144387ad987118275684bf"
format_stamp: "Formatted at 2026-06-18 22:29:25 on dist-test-slave-jm5g"
server_key: "b167169262307d942d3f09c913c56938"
server_key_iv: "e03bdad7c7d2d92b4c963c595d8ac3ac"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:25.700423 29680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:25.749858 29680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:25.754185 29680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:25.754455 29680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:25.755254 29680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:25.756340 29680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:25.756446 29680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:25.756572 29680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:25.756644 29680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:25.769016 29680 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:39543
I20260618 22:29:25.772866 29680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:25.778128 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 29680
I20260618 22:29:25.778249 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:29:25.778580 17358 external_mini_cluster.cc:1428] Setting key 9b4d3cb8481a57be071523e339ef4312
I20260618 22:29:25.774748 29812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:39543 every 8 connection(s)
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
Jun 18 22:29:25 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:25.804569 29816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:25.777149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.129:60281)
0618 22:29:25.783038 (+ 5889us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:25.783113 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:25.784239 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:25.785253 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:25.785263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:25.785624 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:25.786296 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:25.786314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:25.787511 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:25.787515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:25.788927 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:25.788939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:25.789700 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:25.789705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:25.789928 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:25.794746 (+ 4818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:25.794773 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:25.798132 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:25.800849 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:25.800857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:25.800878 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:25.801306 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:25.801680 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:25.801683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:25.801686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:25.801892 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:25.802430 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:25.802438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:25.802828 (+ 390us) client_negotiation.cc:769] Sending connection context
0618 22:29:25.803054 (+ 226us) client_negotiation.cc:240] Negotiation successful
0618 22:29:25.803431 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5595,"thread_start_us":136,"threads_started":1}
I20260618 22:29:25.805600 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:25.776591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:60281 (local address 127.16.243.190:45257)
0618 22:29:25.784695 (+ 8104us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:25.784701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:25.784715 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:29:25.784753 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:25.784756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:25.784848 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:25.784931 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:25.786468 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:25.787363 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:25.789067 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:25.789416 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:25.789517 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:25.798438 (+ 8921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:25.798444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:25.798447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:25.798484 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:25.800657 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:25.801457 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:25.801461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:25.801463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:25.801568 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:25.802002 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:25.802006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:25.802007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:25.802271 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:25.802369 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:25.805179 (+ 2810us) server_negotiation.cc:299] Negotiation successful
0618 22:29:25.805344 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8017,"thread_start_us":83,"threads_started":1}
I20260618 22:29:25.806728 29813 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:25.807024 29813 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:25.807631 29813 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:25.809537 29599 ts_manager.cc:194] Registered new tserver with Master: 15aacffd2f144387ad987118275684bf (127.16.243.129:39543)
I20260618 22:29:25.811254 29599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:60281
Jun 18 22:29:25 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:29:25.823140 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:25.988150 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:25.988646 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:25.988793 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:25.995216 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:25.995384 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:25.995544 29819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:26.004065 29819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:26.005945 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:26.007521 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:26.022797 29828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.023792 29825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.026796 29826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:26.028185 29819 server_base.cc:1034] running on GCE node
I20260618 22:29:26.028801 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:26.029806 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:26.033754 29819 hybrid_clock.cc:648] HybridClock initialized: now 1781821766033597 us; error 160 us; skew 500 ppm
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:26.038421 29819 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:26.039851 29819 webserver.cc:460] Webserver started at http://127.16.243.130:39043/ using document root <none> and password file <none>
I20260618 22:29:26.043799 29819 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:26.043995 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:26.044323 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:26.064265 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "30dbf18b1f4640c6b77a8ba541884f06"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "da0e9e8ffdae45e729eb52e8391c2fe5"
server_key_iv: "0dc21858cf49256cc8ca4b99db0c4091"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.065184 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "30dbf18b1f4640c6b77a8ba541884f06"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "da0e9e8ffdae45e729eb52e8391c2fe5"
server_key_iv: "0dc21858cf49256cc8ca4b99db0c4091"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.071163 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260618 22:29:26.102295 29835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.104564 29819 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260618 22:29:26.104777 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "30dbf18b1f4640c6b77a8ba541884f06"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "da0e9e8ffdae45e729eb52e8391c2fe5"
server_key_iv: "0dc21858cf49256cc8ca4b99db0c4091"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.104954 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:26.134968 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:26.138646 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:26.138914 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:26.139631 29819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:26.140667 29819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:26.140714 29819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.140763 29819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:26.140791 29819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.151569 29819 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45269
I20260618 22:29:26.151676 29948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45269 every 8 connection(s)
I20260618 22:29:26.152848 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:26.154762 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 29819
I20260618 22:29:26.154884 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:29:26.155211 17358 external_mini_cluster.cc:1428] Setting key f024b4a5d7846fcd03c178c2133605cf
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:29:26.187865 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:26.155085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:42707 (local address 127.16.243.190:45257)
0618 22:29:26.155248 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:26.155252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:26.160193 (+ 4941us) server_negotiation.cc:407] Connection header received
0618 22:29:26.161522 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:26.161526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:26.161605 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:26.161680 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:26.163718 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.164542 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.166462 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.166879 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.166990 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:26.170047 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:26.170051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:26.170054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:26.170084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:26.171968 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:26.172682 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:26.172686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:26.172688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:26.172816 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:26.185342 (+ 12526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:26.185347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:26.185349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:26.185657 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:26.185866 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:26.187386 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0618 22:29:26.187569 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:29:26.190152 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:26.155542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.130:42707)
0618 22:29:26.159986 (+ 4444us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:26.160055 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:26.161138 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:26.162290 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:26.162300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:26.162739 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:26.163509 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:26.163534 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.164732 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.164735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:26.166273 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:26.166287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.167180 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.167184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:26.167407 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:26.168035 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:26.168056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:26.169825 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:26.172104 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:26.172112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:26.172134 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:26.172554 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:26.172920 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:26.172923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:26.172925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:26.185036 (+ 12111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:26.185978 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:26.185987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:26.186957 (+ 970us) client_negotiation.cc:769] Sending connection context
0618 22:29:26.188572 (+ 1615us) client_negotiation.cc:240] Negotiation successful
0618 22:29:26.189022 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4187,"thread_start_us":150,"threads_started":1}
I20260618 22:29:26.192063 29949 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:26.192327 29949 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:26.192937 29949 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:26.194175 29599 ts_manager.cc:194] Registered new tserver with Master: 30dbf18b1f4640c6b77a8ba541884f06 (127.16.243.130:45269)
I20260618 22:29:26.195042 29599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:42707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:29:26.201283 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:26.377799 29954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:26.378199 29954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:26.378273 29954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:26.384002 29954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:26.384091 29954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:26.384196 29954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:26.392678 29954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:26.394105 29954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:26.395501 29954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:26.413143 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.413488 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.416129 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:26.416263 29954 server_base.cc:1034] running on GCE node
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821765, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:26.416885 29954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:26.417704 29954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:26.418879 29954 hybrid_clock.cc:648] HybridClock initialized: now 1781821766418812 us; error 80 us; skew 500 ppm
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:26.422644 29954 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:26.423823 29954 webserver.cc:460] Webserver started at http://127.16.243.131:41287/ using document root <none> and password file <none>
I20260618 22:29:26.424834 29954 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:26.424983 29954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:26.425266 29954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:26.443595 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "594a4a71d9824081912477c8db28e36b"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "a427dcd28692277e75b1109329804294"
server_key_iv: "d449b241462915da3a6ca39e18b61bdf"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.445361 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "594a4a71d9824081912477c8db28e36b"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "a427dcd28692277e75b1109329804294"
server_key_iv: "d449b241462915da3a6ca39e18b61bdf"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.450778 29954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260618 22:29:26.472023 29969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.473796 29954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:29:26.473940 29954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "594a4a71d9824081912477c8db28e36b"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "a427dcd28692277e75b1109329804294"
server_key_iv: "d449b241462915da3a6ca39e18b61bdf"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.474069 29954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:26.500196 29954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:26.503953 29954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:26.504204 29954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:26.505088 29954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:26.506345 29954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:26.506462 29954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.506597 29954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:26.506667 29954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.522845 29954 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35357
I20260618 22:29:26.529203 29954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:26.531150 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 29954
I20260618 22:29:26.531273 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:29:26.531605 17358 external_mini_cluster.cc:1428] Setting key 8e0df6f8acb80d545f9b3ab903aa68be
I20260618 22:29:26.530022 30082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35357 every 8 connection(s)
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:26.554211 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:26.536418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:57679 (local address 127.16.243.190:45257)
0618 22:29:26.536556 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:26.536561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:26.539172 (+ 2611us) server_negotiation.cc:407] Connection header received
0618 22:29:26.540547 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:26.540551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:26.540614 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:26.540670 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:26.542481 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.543314 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.545153 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.545515 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.545599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:26.549562 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:26.549566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:26.549570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:26.549600 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:26.551436 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:26.552168 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:26.552173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:26.552175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:26.552276 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:26.552768 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:26.552772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:26.552774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:26.553022 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:26.553102 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:26.553884 (+ 782us) server_negotiation.cc:299] Negotiation successful
0618 22:29:26.554026 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:29:26.556397 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:26.535184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.131:57679)
0618 22:29:26.538975 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:26.539047 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:26.540230 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:26.541157 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:26.541169 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:26.541566 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:26.542287 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:26.542307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.543522 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.543526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:26.544996 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:26.545008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:26.545841 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:26.545845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:26.546049 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:26.547138 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:26.547162 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:26.549347 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:26.551600 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:26.551606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:26.551627 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:26.552038 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:26.552399 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:26.552402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:26.552404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:26.552613 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:26.553191 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:26.553198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:26.553543 (+ 345us) client_negotiation.cc:769] Sending connection context
0618 22:29:26.554752 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0618 22:29:26.555133 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3515,"thread_start_us":192,"threads_started":1}
I20260618 22:29:26.558156 30083 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:26.558418 30083 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:26.559087 30083 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:26.561328 29599 ts_manager.cc:194] Registered new tserver with Master: 594a4a71d9824081912477c8db28e36b (127.16.243.131:35357)
I20260618 22:29:26.562112 29599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:57679
I20260618 22:29:26.563750 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:29:26.564608 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 29680
I20260618 22:29:26.575325 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 29819
I20260618 22:29:26.582098 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 29954
I20260618 22:29:26.589108 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 29567
I20260618 22:29:26.596870 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45257
--webserver_interface=127.16.243.190
--webserver_port=35553
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:45257
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:26.716805 30086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:26.717077 30086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:26.717116 30086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:26.721160 30086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:26.721235 30086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:26.721253 30086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:26.721268 30086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:26.721282 30086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:26.726874 30086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:45257
--ranger_config_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:26.727919 30086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:26.728884 30086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:26.737098 30092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.737098 30091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:26.737170 30086 server_base.cc:1034] running on GCE node
W20260618 22:29:26.737093 30094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:26.737921 30086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:26.739132 30086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:26.740295 30086 hybrid_clock.cc:648] HybridClock initialized: now 1781821766740277 us; error 40 us; skew 500 ppm
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:26.743150 30086 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:26.744063 30086 webserver.cc:460] Webserver started at http://127.16.243.190:35553/ using document root <none> and password file <none>
I20260618 22:29:26.744572 30086 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:26.744639 30086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:29:26 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821766, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:26.793072 30086 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:29:26.795497 30102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:26.796862 30086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:26.797019 30086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "4442a721287c4edaa996c7da6d6162b8"
format_stamp: "Formatted at 2026-06-18 22:29:24 on dist-test-slave-jm5g"
server_key: "1825218294cf9a5e23b6069cf0dae96d"
server_key_iv: "26e13601794ad670eecf861350921bb6"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:26.797588 30086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:26.812670 30086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:26.815933 30086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:26.816140 30086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:26.823962 30086 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:45257
I20260618 22:29:26.824007 30154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:45257 every 8 connection(s)
I20260618 22:29:26.825173 30086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:26.829674 30155 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:29:26.831390 30155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Bootstrap starting.
I20260618 22:29:26.834642 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30086
I20260618 22:29:26.835131 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:39543
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:45257
--webserver_port=41079
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:29:26.847865 30155 log.cc:824] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:26.854499 30155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:29:26.854920 30155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Bootstrap complete.
I20260618 22:29:26.858970 30155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:26.859395 30155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4442a721287c4edaa996c7da6d6162b8, State: Initialized, Role: FOLLOWER
I20260618 22:29:26.859865 30155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:26.859997 30155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:26.860059 30155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:26.860152 30155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:29:26.862110 30155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:26.862429 30155 leader_election.cc:302] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4442a721287c4edaa996c7da6d6162b8; no voters:
I20260618 22:29:26.862757 30155 leader_election.cc:288] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:29:26.862862 30159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:29:26.863056 30159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [term 2 LEADER]: Becoming Leader. State: Replica: 4442a721287c4edaa996c7da6d6162b8, State: Running, Role: LEADER
I20260618 22:29:26.863446 30159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } }
I20260618 22:29:26.864051 30155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:26.864399 30159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4442a721287c4edaa996c7da6d6162b8. Latest consensus state: current_term: 2 leader_uuid: "4442a721287c4edaa996c7da6d6162b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } } }
I20260618 22:29:26.864584 30159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:26.864787 30160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4442a721287c4edaa996c7da6d6162b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4442a721287c4edaa996c7da6d6162b8" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45257 } } }
I20260618 22:29:26.864974 30160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:26.864990 30163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:26.869060 30163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:26.869577 30155 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:29:26.869794 30163 catalog_manager.cc:1259] Loaded cluster ID: 403e76e2f30647d5b05f242c74ca1911
I20260618 22:29:26.869848 30163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:26.872324 30163 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:26.873030 30163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Loaded TSK: 0
I20260618 22:29:26.874254 30163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:29:26.972164 30157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:26.972441 30157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:26.972509 30157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:26.976756 30157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:26.976843 30157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:26.976939 30157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:26.984491 30157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=41079
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:26.986052 30157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:26.987396 30157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:26.997246 30192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.997537 30195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:26.997277 30193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:26.998126 30157 server_base.cc:1034] running on GCE node
I20260618 22:29:26.998590 30157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:26.999316 30157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:27.000514 30157 hybrid_clock.cc:648] HybridClock initialized: now 1781821767000480 us; error 43 us; skew 500 ppm
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:27.004501 30157 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:27.005760 30157 webserver.cc:460] Webserver started at http://127.16.243.129:41079/ using document root <none> and password file <none>
I20260618 22:29:27.006520 30157 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:27.006662 30157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:27.029141 30157 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260618 22:29:27.031785 30203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.033273 30157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:29:27.033474 30157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "15aacffd2f144387ad987118275684bf"
format_stamp: "Formatted at 2026-06-18 22:29:25 on dist-test-slave-jm5g"
server_key: "b167169262307d942d3f09c913c56938"
server_key_iv: "e03bdad7c7d2d92b4c963c595d8ac3ac"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:27.034106 30157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:27.055581 30157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:27.059062 30157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:27.059283 30157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:27.059878 30157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:27.060755 30157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:27.060801 30157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.060839 30157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:27.060868 30157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.070840 30157 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:39543
I20260618 22:29:27.071152 30316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:39543 every 8 connection(s)
I20260618 22:29:27.072214 30157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:27.075181 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30157
I20260618 22:29:27.075647 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:45269
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:45257
--webserver_port=39043
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:27.107357 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.075681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.129:46925)
0618 22:29:27.078603 (+ 2922us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:27.078669 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:27.079905 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:27.082157 (+ 2252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:27.082167 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:27.082576 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:27.083306 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.083331 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.085348 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.085352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.086815 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.086828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.088293 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.088297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.088499 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.089247 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:27.089269 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:27.091136 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:27.093809 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.093816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.093834 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.094234 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.094713 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.094716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.094718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.094914 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.095756 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:27.095763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:27.105544 (+ 9781us) client_negotiation.cc:769] Sending connection context
0618 22:29:27.105855 (+ 311us) client_negotiation.cc:240] Negotiation successful
0618 22:29:27.106261 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"thread_start_us":151,"threads_started":1}
I20260618 22:29:27.109398 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.077394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:46925 (local address 127.16.243.190:45257)
0618 22:29:27.077978 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:27.078002 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:27.078956 (+ 954us) server_negotiation.cc:407] Connection header received
0618 22:29:27.080571 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:27.080587 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:27.081181 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:27.081608 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:27.083572 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.085166 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.086992 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.087752 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.088033 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.091435 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:27.091442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:27.091462 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:27.091499 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:27.093646 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.094390 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.094397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.094403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.094600 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.095027 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.095031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.095032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.095561 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:27.095698 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:27.108054 (+ 12356us) server_negotiation.cc:299] Negotiation successful
0618 22:29:27.108415 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":118,"threads_started":1}
I20260618 22:29:27.113317 30317 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:27.113579 30317 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.114188 30317 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.116597 30317 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45257 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:29:27.116721 30317 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.116935 30317 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:27.117916 30317 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.118108 30317 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.118950 30317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:29:27.225354 30320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:27.225749 30320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:27.225819 30320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:27.231947 30320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:27.232036 30320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:27.232127 30320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:27.237959 30320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:45269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=39043
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:27.239256 30320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:27.240336 30320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:27.254056 30329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:27.254066 30326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:27.254667 30327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:27.254739 30320 server_base.cc:1034] running on GCE node
I20260618 22:29:27.255165 30320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:27.255805 30320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:27.256956 30320 hybrid_clock.cc:648] HybridClock initialized: now 1781821767256923 us; error 56 us; skew 500 ppm
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:27.260169 30320 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:27.261162 30320 webserver.cc:460] Webserver started at http://127.16.243.130:39043/ using document root <none> and password file <none>
I20260618 22:29:27.261725 30320 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:27.261798 30320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:27.279497 30320 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:29:27.282543 30337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.284039 30320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:29:27.284188 30320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "30dbf18b1f4640c6b77a8ba541884f06"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "da0e9e8ffdae45e729eb52e8391c2fe5"
server_key_iv: "0dc21858cf49256cc8ca4b99db0c4091"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:27.284765 30320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:27.320417 30320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:27.324613 30320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:27.324797 30320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:27.325490 30320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:27.326457 30320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:27.326499 30320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.326581 30320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:27.326606 30320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.335522 30320 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45269
I20260618 22:29:27.335773 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45269 every 8 connection(s)
I20260618 22:29:27.336809 30320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:27.345165 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30320
I20260618 22:29:27.345635 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.243.148:39513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:35357
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:45257
--webserver_port=41287
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:39255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:27.355078 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.339896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.130:39929)
0618 22:29:27.340467 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:27.340521 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:27.341358 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:27.342080 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:27.342089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:27.342454 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:27.343174 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.343194 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.344432 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.344435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.345903 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.345914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.346727 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.346732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.346943 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.347644 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:27.347665 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:27.349397 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:27.351597 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.351604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.351624 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.352007 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.352326 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.352329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.352331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.352526 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.353035 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:27.353042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:27.353389 (+ 347us) client_negotiation.cc:769] Sending connection context
0618 22:29:27.353602 (+ 213us) client_negotiation.cc:240] Negotiation successful
0618 22:29:27.353969 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":149,"threads_started":1}
I20260618 22:29:27.355762 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.339230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39929 (local address 127.16.243.190:45257)
0618 22:29:27.339374 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:27.339378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:27.340634 (+ 1256us) server_negotiation.cc:407] Connection header received
0618 22:29:27.341576 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:27.341581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:27.341640 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:27.341752 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:27.343376 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.344262 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.346053 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.346391 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.346474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.349617 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:27.349622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:27.349625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:27.349657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:27.351441 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.352132 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.352135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.352137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.352228 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.352637 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.352641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.352643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.352887 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:27.352981 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:27.355473 (+ 2492us) server_negotiation.cc:299] Negotiation successful
0618 22:29:27.355610 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:29:27.357470 30451 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:27.357731 30451 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.358294 30451 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.360036 30451 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45257 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:29:27.360152 30451 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.360312 30451 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:27.361155 30451 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.361311 30451 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.362046 30451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:29:27.510423 30454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:27.510893 30454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:27.510977 30454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:27.516974 30454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:27.517066 30454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:27.517154 30454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:27.525134 30454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:39255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.243.148:39513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:35357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=41287
--tserver_master_addrs=127.16.243.190:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:27.526604 30454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:27.528005 30454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:27.541019 30460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:27.541050 30463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:27.541607 30461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:27.542173 30454 server_base.cc:1034] running on GCE node
I20260618 22:29:27.542829 30454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:27.543704 30454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:27.544898 30454 hybrid_clock.cc:648] HybridClock initialized: now 1781821767544862 us; error 48 us; skew 500 ppm
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:27.548468 30454 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:27.549666 30454 webserver.cc:460] Webserver started at http://127.16.243.131:41287/ using document root <none> and password file <none>
I20260618 22:29:27.550446 30454 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:27.550628 30454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:27.579244 30454 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260618 22:29:27.582052 30471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.583479 30454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:27.583604 30454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "594a4a71d9824081912477c8db28e36b"
format_stamp: "Formatted at 2026-06-18 22:29:26 on dist-test-slave-jm5g"
server_key: "a427dcd28692277e75b1109329804294"
server_key_iv: "d449b241462915da3a6ca39e18b61bdf"
server_key_version: "kuduclusterkey@0"
I20260618 22:29:27.584220 30454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:27.599844 30454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:27.604349 30454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:27.604666 30454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:27.605536 30454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:27.606637 30454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:27.606700 30454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.606756 30454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:27.606798 30454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:27.619750 30454 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35357
I20260618 22:29:27.620955 30454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:27.621516 30584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35357 every 8 connection(s)
I20260618 22:29:27.628615 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30454
Jun 18 22:29:27 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821767, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:27.637171 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.623096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:54467 (local address 127.16.243.190:45257)
0618 22:29:27.623226 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:27.623230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:27.624455 (+ 1225us) server_negotiation.cc:407] Connection header received
0618 22:29:27.625710 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:27.625713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:27.625772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:27.625849 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:27.627044 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.627879 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.629510 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.629825 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.629905 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.632996 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:27.633000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:27.633003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:27.633033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:27.634698 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.635308 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.635311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.635313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.635397 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.635853 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.635857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.635859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.636094 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:27.636168 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:27.636899 (+ 731us) server_negotiation.cc:299] Negotiation successful
0618 22:29:27.637014 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:29:27.639117 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.623684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45257 (local address 127.16.243.131:54467)
0618 22:29:27.624279 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:27.624340 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:27.625502 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:27.626034 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:27.626041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:27.626311 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:27.626869 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.626889 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.628041 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.628044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.629377 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:27.629389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.630103 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.630107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:27.630308 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.630978 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:27.630999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:27.632754 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:27.634829 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.634836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.634855 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.635204 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.635512 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:27.635515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:27.635517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:27.635709 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:27.636256 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:27.636262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:27.636609 (+ 347us) client_negotiation.cc:769] Sending connection context
0618 22:29:27.637635 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0618 22:29:27.637993 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260618 22:29:27.640697 30585 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45257
I20260618 22:29:27.641013 30585 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.641637 30585 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.643391 30585 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45257 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:29:27.643580 30585 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.643798 30585 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:27.644615 30585 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:27.644826 30585 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
W20260618 22:29:27.645520 30585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:29:27.649356 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:27.629607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.16.243.190:45257)
0618 22:29:27.634707 (+ 5100us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:27.634716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:27.634728 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:29:27.634915 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:27.634921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:27.634979 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:27.635062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:27.636385 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.637096 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.638214 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:27.638579 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:27.638669 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:27.642536 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:27.642540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:27.642542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:27.642588 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:27.647341 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.647786 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.647790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.647792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.647872 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:27.648361 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:27.648365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:27.648366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:27.648558 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:27.648748 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:27.648942 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:29:27.649063 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5028,"thread_start_us":78,"threads_started":1}
Jun 18 22:29:28 dist-test-slave-jm5g krb5kdc[29123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:28.119186 30317 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:28.119488 30317 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:28.120784 30119 ts_manager.cc:194] Registered new tserver with Master: 15aacffd2f144387ad987118275684bf (127.16.243.129:39543)
I20260618 22:29:28.122485 30119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:46925
I20260618 22:29:28.362264 30451 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:28.362608 30451 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:28.363495 30119 ts_manager.cc:194] Registered new tserver with Master: 30dbf18b1f4640c6b77a8ba541884f06 (127.16.243.130:45269)
I20260618 22:29:28.364267 30119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:39929
Jun 18 22:29:28 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:28.645747 30585 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:28.646073 30585 heartbeater.cc:511] Master 127.16.243.190:45257 requested a full tablet report, sending...
I20260618 22:29:28.646960 30119 ts_manager.cc:194] Registered new tserver with Master: 594a4a71d9824081912477c8db28e36b (127.16.243.131:35357)
I20260618 22:29:28.647729 30119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:54467
I20260618 22:29:28.659345 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:29:28 dist-test-slave-jm5g krb5kdc[29123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821768, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for HTTP/127.16.243.148@KRBTEST.COM
I20260618 22:29:28.677968 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.661725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.16.243.190:45257)
0618 22:29:28.665478 (+ 3753us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.665483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.665825 (+ 342us) server_negotiation.cc:407] Connection header received
0618 22:29:28.666050 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.666054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.666109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.666226 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:28.667244 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.668088 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.669252 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.669604 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.669687 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.671677 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:28.671681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:28.671684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:28.671714 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:28.673786 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:28.674436 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:28.674440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:28.674442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:28.674534 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:28.675082 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:28.675085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:28.675087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:28.675338 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:28.675455 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.677617 (+ 2162us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.677749 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3652,"thread_start_us":138,"threads_started":1}
I20260618 22:29:28.690011 30119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:29:28.693697 30119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:29:28.723953 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.712301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45269 (local address 127.0.0.1:41848)
0618 22:29:28.715839 (+ 3538us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.716133 (+ 294us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:28.716225 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:28.717530 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:28.717533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:28.717550 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:28.717988 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.717995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.720105 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.720108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.721128 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.721136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.723338 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.723342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.723486 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.723540 (+ 54us) client_negotiation.cc:769] Sending connection context
0618 22:29:28.723644 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 22:29:28.723780 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3464,"thread_start_us":74,"threads_started":1}
I20260618 22:29:28.724973 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.712813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.16.243.129:39543)
0618 22:29:28.713070 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.713077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.714017 (+ 940us) server_negotiation.cc:407] Connection header received
0618 22:29:28.714245 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.714251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.714449 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.714587 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:28.715428 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.718134 (+ 2706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.722030 (+ 3896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.723733 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.723824 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.723957 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.724720 (+ 763us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.724833 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260618 22:29:28.725215 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.711997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39543 (local address 127.0.0.1:50022)
0618 22:29:28.713918 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.713932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:28.714091 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:28.715132 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:28.715139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:28.715165 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:28.715312 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.715319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.718404 (+ 3085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.718410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.719983 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.719994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.724052 (+ 4058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.724056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.724146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.724164 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 22:29:28.724220 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:29:28.724286 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"thread_start_us":98,"threads_started":1}
I20260618 22:29:28.731802 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.709356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35357 (local address 127.0.0.1:50912)
0618 22:29:28.712509 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.712570 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:28.712705 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:28.727353 (+ 14648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:28.727357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:28.727378 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:28.727480 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.727485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.729057 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.729061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.730200 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.730211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.731386 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.731390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.731481 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.731511 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:29:28.731562 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 22:29:28.731642 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2913,"thread_start_us":113,"threads_started":1}
I20260618 22:29:28.732685 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.710496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.16.243.131:35357)
0618 22:29:28.726788 (+ 16292us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.726796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.726825 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:29:28.726920 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.726926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.727177 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.727633 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:28.727788 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.728919 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.730355 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.731248 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.732235 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.732392 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.732451 (+ 59us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.732552 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16162,"thread_start_us":133,"threads_started":1}
I20260618 22:29:28.733665 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.713136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.16.243.130:45269)
0618 22:29:28.714208 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.714217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.716536 (+ 2319us) server_negotiation.cc:407] Connection header received
0618 22:29:28.716987 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.716993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.717225 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.717355 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:28.718321 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.719768 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.721481 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.722828 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.733182 (+ 10354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.733364 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.733424 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.733528 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260618 22:29:28.735471 30251 tablet_service.cc:1467] Processing CreateTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e (DEFAULT_TABLE table=test-table [id=7802092cdd6246f195a2f4865ae80adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:28.735570 30500 tablet_service.cc:1467] Processing CreateTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e (DEFAULT_TABLE table=test-table [id=7802092cdd6246f195a2f4865ae80adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:28.736459 30251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13551a12cb49464ea3f4a07e9b5d0a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:28.736572 30500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13551a12cb49464ea3f4a07e9b5d0a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:28.741035 30381 tablet_service.cc:1467] Processing CreateTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e (DEFAULT_TABLE table=test-table [id=7802092cdd6246f195a2f4865ae80adb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:28.742012 30381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13551a12cb49464ea3f4a07e9b5d0a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:28.753440 30620 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Bootstrap starting.
I20260618 22:29:28.755906 30620 tablet_bootstrap.cc:654] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:28.756806 30620 log.cc:824] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:28.757650 30621 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Bootstrap starting.
I20260618 22:29:28.760160 30621 tablet_bootstrap.cc:654] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:28.763859 30620 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: No bootstrap required, opened a new log
I20260618 22:29:28.764635 30620 ts_tablet_manager.cc:1397] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260618 22:29:28.764964 30621 log.cc:824] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:28.765825 30622 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Bootstrap starting.
I20260618 22:29:28.768361 30622 tablet_bootstrap.cc:654] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:28.769281 30622 log.cc:824] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:28.772324 30620 raft_consensus.cc:348] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.772655 30620 raft_consensus.cc:374] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:28.772720 30620 raft_consensus.cc:729] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dbf18b1f4640c6b77a8ba541884f06, State: Initialized, Role: FOLLOWER
I20260618 22:29:28.773277 30620 consensus_queue.cc:260] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.776010 30620 ts_tablet_manager.cc:1428] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260618 22:29:28.776383 30451 heartbeater.cc:503] Master 127.16.243.190:45257 was elected leader, sending a full tablet report...
I20260618 22:29:28.777133 30621 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: No bootstrap required, opened a new log
I20260618 22:29:28.777377 30621 ts_tablet_manager.cc:1397] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260618 22:29:28.781373 30621 raft_consensus.cc:348] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.781684 30621 raft_consensus.cc:374] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:28.781739 30621 raft_consensus.cc:729] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594a4a71d9824081912477c8db28e36b, State: Initialized, Role: FOLLOWER
I20260618 22:29:28.782218 30621 consensus_queue.cc:260] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.791618 30621 ts_tablet_manager.cc:1428] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260618 22:29:28.792037 30585 heartbeater.cc:503] Master 127.16.243.190:45257 was elected leader, sending a full tablet report...
I20260618 22:29:28.792959 30622 tablet_bootstrap.cc:492] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: No bootstrap required, opened a new log
I20260618 22:29:28.793258 30622 ts_tablet_manager.cc:1397] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260618 22:29:28.797947 30622 raft_consensus.cc:348] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.798393 30622 raft_consensus.cc:374] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:28.798501 30622 raft_consensus.cc:729] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aacffd2f144387ad987118275684bf, State: Initialized, Role: FOLLOWER
I20260618 22:29:28.799155 30622 consensus_queue.cc:260] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.800431 30622 ts_tablet_manager.cc:1428] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260618 22:29:28.800854 30317 heartbeater.cc:503] Master 127.16.243.190:45257 was elected leader, sending a full tablet report...
I20260618 22:29:28.818717 30626 raft_consensus.cc:482] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:28.818957 30626 raft_consensus.cc:504] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.820377 30626 leader_election.cc:288] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 594a4a71d9824081912477c8db28e36b (127.16.243.131:35357), 15aacffd2f144387ad987118275684bf (127.16.243.129:39543)
I20260618 22:29:28.835064 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.821096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:46061 (local address 127.16.243.129:39543)
0618 22:29:28.821219 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.821224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.830844 (+ 9620us) server_negotiation.cc:407] Connection header received
0618 22:29:28.831031 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.831034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.831080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.831123 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:28.831514 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.832234 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.833727 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.834343 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.834440 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.834492 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.834842 (+ 350us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.834926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:29:28.835546 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.820699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39543 (local address 127.16.243.130:46061)
0618 22:29:28.830748 (+ 10049us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.830764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:28.830921 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:28.831246 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:28.831249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:28.831264 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:28.831397 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.831402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.832386 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.832389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.833579 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.833589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.834608 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.834611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.834713 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.834737 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:29:28.835293 (+ 556us) client_negotiation.cc:240] Negotiation successful
0618 22:29:28.835388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9962,"thread_start_us":103,"threads_started":1}
W20260618 22:29:28.835893 30318 tablet.cc:2369] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:28.836210 30271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13551a12cb49464ea3f4a07e9b5d0a0e" candidate_uuid: "30dbf18b1f4640c6b77a8ba541884f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15aacffd2f144387ad987118275684bf" is_pre_election: true
I20260618 22:29:28.836508 30271 raft_consensus.cc:2454] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30dbf18b1f4640c6b77a8ba541884f06 in term 0.
I20260618 22:29:28.837049 30340 leader_election.cc:302] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15aacffd2f144387ad987118275684bf, 30dbf18b1f4640c6b77a8ba541884f06; no voters:
I20260618 22:29:28.837381 30626 raft_consensus.cc:2781] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:28.837463 30626 raft_consensus.cc:482] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:28.837502 30626 raft_consensus.cc:3037] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:29:28.838835 30452 tablet.cc:2369] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:28.838819 30626 raft_consensus.cc:504] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.839237 30626 leader_election.cc:288] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [CANDIDATE]: Term 1 election: Requested vote from peers 594a4a71d9824081912477c8db28e36b (127.16.243.131:35357), 15aacffd2f144387ad987118275684bf (127.16.243.129:39543)
I20260618 22:29:28.840091 30271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13551a12cb49464ea3f4a07e9b5d0a0e" candidate_uuid: "30dbf18b1f4640c6b77a8ba541884f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15aacffd2f144387ad987118275684bf"
I20260618 22:29:28.840229 30271 raft_consensus.cc:3037] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:28.841526 30271 raft_consensus.cc:2454] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30dbf18b1f4640c6b77a8ba541884f06 in term 1.
I20260618 22:29:28.841998 30340 leader_election.cc:302] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15aacffd2f144387ad987118275684bf, 30dbf18b1f4640c6b77a8ba541884f06; no voters:
I20260618 22:29:28.842257 30626 raft_consensus.cc:2781] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:28.842469 30626 raft_consensus.cc:686] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 1 LEADER]: Becoming Leader. State: Replica: 30dbf18b1f4640c6b77a8ba541884f06, State: Running, Role: LEADER
I20260618 22:29:28.842968 30626 consensus_queue.cc:237] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } }
I20260618 22:29:28.846673 30119 catalog_manager.cc:5495] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30dbf18b1f4640c6b77a8ba541884f06 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "30dbf18b1f4640c6b77a8ba541884f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30dbf18b1f4640c6b77a8ba541884f06" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45269 } health_report { overall_health: HEALTHY } } }
I20260618 22:29:28.851253 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.821127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:55265 (local address 127.16.243.131:35357)
0618 22:29:28.821247 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.821251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.826922 (+ 5671us) server_negotiation.cc:407] Connection header received
0618 22:29:28.827137 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.827140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.827189 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.827233 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:28.827661 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.828423 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.838647 (+ 10224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.839243 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.839380 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.839437 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.851001 (+ 11564us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.851111 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:29:28.851711 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.820622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35357 (local address 127.16.243.130:55265)
0618 22:29:28.826786 (+ 6164us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.826806 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:28.827017 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:28.827383 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:28.827386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:28.827403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:28.827546 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.827551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.828577 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.828581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.829917 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:28.829930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.850652 (+ 20722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.850656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:28.850792 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.850825 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:29:28.851468 (+ 643us) client_negotiation.cc:240] Negotiation successful
0618 22:29:28.851564 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6065,"thread_start_us":136,"threads_started":1}
I20260618 22:29:28.852396 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13551a12cb49464ea3f4a07e9b5d0a0e" candidate_uuid: "30dbf18b1f4640c6b77a8ba541884f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594a4a71d9824081912477c8db28e36b" is_pre_election: true
I20260618 22:29:28.852732 30539 raft_consensus.cc:2454] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30dbf18b1f4640c6b77a8ba541884f06 in term 0.
I20260618 22:29:28.853056 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13551a12cb49464ea3f4a07e9b5d0a0e" candidate_uuid: "30dbf18b1f4640c6b77a8ba541884f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594a4a71d9824081912477c8db28e36b"
I20260618 22:29:28.853170 30537 raft_consensus.cc:3037] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:28.854528 30537 raft_consensus.cc:2454] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30dbf18b1f4640c6b77a8ba541884f06 in term 1.
W20260618 22:29:28.873888 30586 tablet.cc:2369] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:28.876837 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.871850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.16.243.130:45269)
0618 22:29:28.871968 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:28.871972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:28.871986 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:29:28.872080 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:28.872083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:28.872145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:28.872247 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:29:28.872602 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.873376 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.874649 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:28.874999 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:28.875091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:28.875506 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:29:28.876062 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:29:28.876170 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:28.876595 (+ 425us) server_negotiation.cc:299] Negotiation successful
0618 22:29:28.876660 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:29:28.896502 30271 raft_consensus.cc:1261] T 13551a12cb49464ea3f4a07e9b5d0a0e P 15aacffd2f144387ad987118275684bf [term 1 FOLLOWER]: Refusing update from remote peer 30dbf18b1f4640c6b77a8ba541884f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:28.897557 30537 raft_consensus.cc:1261] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 1 FOLLOWER]: Refusing update from remote peer 30dbf18b1f4640c6b77a8ba541884f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:28.898727 30626 consensus_queue.cc:1035] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15aacffd2f144387ad987118275684bf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:28.898967 30631 consensus_queue.cc:1035] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "594a4a71d9824081912477c8db28e36b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:28.917192 30634 mvcc.cc:205] Tried to move back new op lower bound from 7298341965369090048 to 7298341965184888832. Current Snapshot: MvccSnapshot[applied={T|T < 7298341965369090048}]
I20260618 22:29:28.920845 30635 mvcc.cc:205] Tried to move back new op lower bound from 7298341965369090048 to 7298341965184888832. Current Snapshot: MvccSnapshot[applied={T|T < 7298341965369090048}]
I20260618 22:29:28.931453 30119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:28.931655 30119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:28.934459 30119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4442a721287c4edaa996c7da6d6162b8: Sending DeleteTablet for 3 replicas of tablet 13551a12cb49464ea3f4a07e9b5d0a0e
I20260618 22:29:28.935638 30251 tablet_service.cc:1514] Processing DeleteTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:28 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:50022
I20260618 22:29:28.936550 30500 tablet_service.cc:1514] Processing DeleteTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:28 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:50912
I20260618 22:29:28.937377 30381 tablet_service.cc:1514] Processing DeleteTablet for tablet 13551a12cb49464ea3f4a07e9b5d0a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:28 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:41848
I20260618 22:29:28.938470 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30157
I20260618 22:29:28.939121 30643 tablet_replica.cc:330] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: stopping tablet replica
I20260618 22:29:28.939495 30643 raft_consensus.cc:2229] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:28.939860 30643 raft_consensus.cc:2258] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:28.940235 30644 tablet_replica.cc:330] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: stopping tablet replica
I20260618 22:29:28.940543 30644 raft_consensus.cc:2229] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:29:28.940944 30644 raft_consensus.cc:2258] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:28.942659 30644 ts_tablet_manager.cc:1905] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:28.945566 30644 ts_tablet_manager.cc:1918] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:28.945679 30644 log.cc:1196] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/13551a12cb49464ea3f4a07e9b5d0a0e
I20260618 22:29:28.946012 30644 ts_tablet_manager.cc:1939] T 13551a12cb49464ea3f4a07e9b5d0a0e P 30dbf18b1f4640c6b77a8ba541884f06: Deleting consensus metadata
I20260618 22:29:28.947350 30105 catalog_manager.cc:4841] TS 30dbf18b1f4640c6b77a8ba541884f06 (127.16.243.130:45269): tablet 13551a12cb49464ea3f4a07e9b5d0a0e (table test-table [id=7802092cdd6246f195a2f4865ae80adb]) successfully deleted
W20260618 22:29:28.949124 30340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:39543 (error 108)
I20260618 22:29:28.949127 30643 ts_tablet_manager.cc:1905] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:29:28.950904 30105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:39543 (error 108)
I20260618 22:29:28.953037 30643 ts_tablet_manager.cc:1918] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:28.953161 30643 log.cc:1196] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/13551a12cb49464ea3f4a07e9b5d0a0e
I20260618 22:29:28.953531 30643 ts_tablet_manager.cc:1939] T 13551a12cb49464ea3f4a07e9b5d0a0e P 594a4a71d9824081912477c8db28e36b: Deleting consensus metadata
I20260618 22:29:28.953998 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30320
I20260618 22:29:28.955804 30104 catalog_manager.cc:4841] TS 594a4a71d9824081912477c8db28e36b (127.16.243.131:35357): tablet 13551a12cb49464ea3f4a07e9b5d0a0e (table test-table [id=7802092cdd6246f195a2f4865ae80adb]) successfully deleted
I20260618 22:29:28.956972 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:28.956642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39543 (local address 127.0.0.1:50032)
0618 22:29:28.956818 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:28.956886 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260618 22:29:28.957147 30105 catalog_manager.cc:4584] TS 15aacffd2f144387ad987118275684bf (127.16.243.129:39543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13551a12cb49464ea3f4a07e9b5d0a0e: Network error: Client connection negotiation failed: client connection to 127.16.243.129:39543: connect: Connection refused (error 111)
I20260618 22:29:28.962747 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30454
I20260618 22:29:28.969067 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30086
I20260618 22:29:28.976366 17358 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:29:29.311188 17358 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:29:29.311295 17358 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:29:29.646330 17358 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:29:29.646 UTC [29153] LOG: received smart shutdown request
2026-06-18 22:29:29.650 UTC [29153] LOG: background worker "logical replication launcher" (PID 29162) exited with exit code 1
2026-06-18 22:29:29.650 UTC [29157] LOG: shutting down
2026-06-18 22:29:29.785 UTC [29153] LOG: database system is shut down
2026-06-18T22:29:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47383 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:29:29 dist-test-slave-jm5g krb5kdc[30655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:29:29 dist-test-slave-jm5g krb5kdc[30655](info): set up 2 sockets
Jun 18 22:29:29 dist-test-slave-jm5g krb5kdc[30655](info): commencing operation
krb5kdc: starting...
W20260618 22:29:30.838376 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:29:30 dist-test-slave-jm5g krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:29:30.954768 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42667
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:31.066304 30668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:31.066649 30668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:31.066717 30668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:31.070819 30668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:31.070894 30668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:31.070922 30668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:31.070941 30668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:31.070972 30668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:31.076325 30668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:31.077450 30668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:31.078495 30668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:31.086900 30676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:31.087061 30673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:31.086882 30674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:31.087178 30668 server_base.cc:1034] running on GCE node
I20260618 22:29:31.087849 30668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:31.088968 30668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:31.090153 30668 hybrid_clock.cc:648] HybridClock initialized: now 1781821771090135 us; error 33 us; skew 500 ppm
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:31.092936 30668 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:31.093825 30668 webserver.cc:460] Webserver started at http://127.16.243.190:43395/ using document root <none> and password file <none>
I20260618 22:29:31.094355 30668 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:31.094419 30668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:31.094655 30668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:31.096681 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "cff57730309c460eaec54afb77e48d4f"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "d76fc65700c199341693727a6298d7c0"
server_key_iv: "162416849dc199b367c28b243c0b6210"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.097240 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "cff57730309c460eaec54afb77e48d4f"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "d76fc65700c199341693727a6298d7c0"
server_key_iv: "162416849dc199b367c28b243c0b6210"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.100972 30668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:29:31.103425 30683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:31.104682 30668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:31.104820 30668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "cff57730309c460eaec54afb77e48d4f"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "d76fc65700c199341693727a6298d7c0"
server_key_iv: "162416849dc199b367c28b243c0b6210"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.104923 30668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:31.125186 30668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:31.128271 30668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:31.128453 30668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:31.135540 30668 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:42667
I20260618 22:29:31.135875 30735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:42667 every 8 connection(s)
I20260618 22:29:31.136523 30668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:31.139312 30736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:31.140892 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30668
I20260618 22:29:31.141004 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:29:31.141253 17358 external_mini_cluster.cc:1428] Setting key fd45ec7d2aebb31e3cb9585048b2fdea
I20260618 22:29:31.143450 30736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.189@KRBTEST.COM" created.
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:31.158782 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.147045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.16.243.190:42667)
0618 22:29:31.147365 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.147375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.147418 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:29:31.147793 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.147797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.147978 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.148139 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.148980 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.149852 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.150866 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.151297 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.151453 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.154004 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.154011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.154024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.154049 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.155769 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.156200 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.156206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.156209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.156312 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.156577 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.156580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.156582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.156967 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.157086 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.157433 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.157713 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260618 22:29:31.158782 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.146003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42667 (local address 127.0.0.1:47274)
0618 22:29:31.146535 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.146625 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.147507 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.148147 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.148152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.148562 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.148847 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.148861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.149966 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.149969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.150746 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.150757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.151462 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.151466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.151574 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.152164 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.152185 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.153852 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.155875 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.155882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.155893 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.156102 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.156393 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.156396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.156398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.156498 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.157071 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.157077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.157201 (+ 124us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.157360 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.157765 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":97,"thread_start_us":144,"threads_started":1}
I20260618 22:29:31.161088 30736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:29:31.161746 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.161506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52116)
0618 22:29:31.161616 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.161681 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:29:31.161953 30687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
I20260618 22:29:31.162746 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.162540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52118)
0618 22:29:31.162649 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.162692 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260618 22:29:31.162868 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:36997: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
I20260618 22:29:31.165227 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:36997
--webserver_interface=127.16.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:29:31.212096 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } attempt: 1
I20260618 22:29:31.212967 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.212674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52134)
0618 22:29:31.212829 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.212892 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 22:29:31.213629 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.213420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52136)
0618 22:29:31.213524 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.213569 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:29:31.213738 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:36997: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
W20260618 22:29:31.276077 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:31.276377 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:31.276419 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:31.280498 30743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:31.280566 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:31.280584 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:31.280601 30743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:31.280614 30743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:31.281890 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } attempt: 2
I20260618 22:29:31.282686 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.282374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52146)
0618 22:29:31.282515 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.282607 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:29:31.283463 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.283214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52162)
0618 22:29:31.283318 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.283396 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:29:31.283640 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:36997: Network error: Client connection negotiation failed: client connection to 127.16.243.189:36997: connect: Connection refused (error 111)
I20260618 22:29:31.286423 30743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:31.287545 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:31.288537 30743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:31.297159 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:31.297243 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:31.297148 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:31.297348 30743 server_base.cc:1034] running on GCE node
I20260618 22:29:31.298017 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:31.299173 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:31.300330 30743 hybrid_clock.cc:648] HybridClock initialized: now 1781821771300314 us; error 34 us; skew 500 ppm
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:31.303023 30743 init.cc:377] Logged in from keytab as kudu/127.16.243.189@KRBTEST.COM (short username kudu)
I20260618 22:29:31.303937 30743 webserver.cc:460] Webserver started at http://127.16.243.189:33855/ using document root <none> and password file <none>
I20260618 22:29:31.304471 30743 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:31.304538 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:31.304723 30743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:31.306824 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data/instance:
uuid: "21366e1a98024de8a61a98b9320b0eb8"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "a90cf4b73143d75a93bfb63c052db576"
server_key_iv: "6a1a7f4155c92c5b936fe071778477a4"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.307379 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal/instance:
uuid: "21366e1a98024de8a61a98b9320b0eb8"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "a90cf4b73143d75a93bfb63c052db576"
server_key_iv: "6a1a7f4155c92c5b936fe071778477a4"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.311271 30743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:29:31.313795 30758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:31.315063 30743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:31.315181 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "21366e1a98024de8a61a98b9320b0eb8"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "a90cf4b73143d75a93bfb63c052db576"
server_key_iv: "6a1a7f4155c92c5b936fe071778477a4"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.315284 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:31.327402 30743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:31.330410 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:31.330637 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:31.337507 30743 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.189:36997
I20260618 22:29:31.337548 30810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.189:36997 every 8 connection(s)
I20260618 22:29:31.338456 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
I20260618 22:29:31.340813 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30743
I20260618 22:29:31.340924 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-1/wal/instance
I20260618 22:29:31.341197 17358 external_mini_cluster.cc:1428] Setting key 8326de9d1b69fd70b9959c162f079f5c
I20260618 22:29:31.341398 30811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:31.345242 30811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.188@KRBTEST.COM" created.
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:31.359522 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.347181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.16.243.190:42667)
0618 22:29:31.347306 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.347309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.348459 (+ 1150us) server_negotiation.cc:407] Connection header received
0618 22:29:31.349413 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.349416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.349480 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.349553 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.350786 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.351299 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.352856 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.353103 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.353199 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.355827 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.355830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.355832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.355861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.357365 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.357891 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.357895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.357896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.357982 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.358312 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.358315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.358316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.358509 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.358642 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.359221 (+ 579us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.359360 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:29:31.360311 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.347737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42667 (local address 127.0.0.1:47284)
0618 22:29:31.348303 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.348350 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.349204 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.349733 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.349740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.350072 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.350647 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.350663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.351421 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.351425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.352748 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.352760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.353263 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.353266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.353446 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.354021 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.354038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.355603 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.357497 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.357504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.357523 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.357779 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.358079 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.358083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.358085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.358230 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.358656 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.358664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.358949 (+ 285us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.359136 (+ 187us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.359430 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":145,"threads_started":1}
I20260618 22:29:31.361797 30811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
I20260618 22:29:31.364709 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:46301
--webserver_interface=127.16.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:46851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:29:31.372215 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.362305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52178)
0618 22:29:31.362432 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.362444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.362515 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.363154 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.363157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.363347 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.363448 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.363454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.364475 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.364479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.365088 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.365098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.365735 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.365738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.365852 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.366330 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.366342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.367658 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.370161 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.370165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.370167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.370591 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.370948 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.370951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.370953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.371072 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.371643 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.371647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.371760 (+ 113us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.371867 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.372025 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":11}
I20260618 22:29:31.372417 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.362442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.16.243.189:36997)
0618 22:29:31.362786 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.362796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.362817 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:29:31.362855 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.362860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.363021 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.363156 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.363566 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.364352 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.365216 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.365613 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.365727 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.367792 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.367802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.367816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.367837 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.369946 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.370722 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.370730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.370732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.370832 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.371167 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.371170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.371172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.371530 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.371674 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.372014 (+ 340us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.372178 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260618 22:29:31.374801 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } attempt: 3
I20260618 22:29:31.375641 30811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:29:31.376428 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.376063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33448)
0618 22:29:31.376249 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.376328 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260618 22:29:31.376605 30761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
I20260618 22:29:31.377461 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.377238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33454)
0618 22:29:31.377336 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.377395 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:29:31.377588 30811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:29:31.384599 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.375315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52180)
0618 22:29:31.375567 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.375581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.375663 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.376096 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.376099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.376290 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.376426 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.376433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.377755 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.377758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.379368 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.379381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.379876 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.379880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.379988 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.380443 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.380455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.381608 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.383175 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.383179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.383181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.383404 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.383650 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.383653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.383655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.383732 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.384088 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.384091 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.384185 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.384270 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.384426 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260618 22:29:31.384598 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.375543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.16.243.189:36997)
0618 22:29:31.375850 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.375867 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.375880 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:29:31.375915 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.375918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.375963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.376571 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.376659 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.377487 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.379481 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.379753 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.379851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.381738 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.381742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.381745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.381768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.383068 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.383494 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.383497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.383498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.383569 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.383809 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.383813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.383815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.384000 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.384084 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.384327 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.384436 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260618 22:29:31.385838 30736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:29:31.386514 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.386272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33456)
0618 22:29:31.386397 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.386450 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:29:31.387136 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.386905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33462)
0618 22:29:31.387030 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.387084 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260618 22:29:31.387238 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
I20260618 22:29:31.418399 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 1
I20260618 22:29:31.419291 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.418985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33466)
0618 22:29:31.419120 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.419208 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:29:31.419950 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.419719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33470)
0618 22:29:31.419813 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.419881 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:29:31.420066 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
I20260618 22:29:31.426848 30811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 1
I20260618 22:29:31.427695 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.427411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33486)
0618 22:29:31.427558 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.427617 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:29:31.428335 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.428139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33494)
0618 22:29:31.428234 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.428277 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:29:31.428447 30811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
W20260618 22:29:31.478909 30817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:31.479195 30817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:31.479250 30817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:31.483330 30817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:31.483431 30817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:31.483457 30817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:31.483481 30817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:31.483510 30817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:31.489001 30817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:46851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42667,127.16.243.189:36997,127.16.243.188:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:31.490126 30817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:31.491262 30817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:29:31.495270 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 2
I20260618 22:29:31.496089 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.495826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33506)
0618 22:29:31.495948 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.496010 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 22:29:31.496604 30811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 2
I20260618 22:29:31.496788 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.496567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33510)
0618 22:29:31.496666 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.496715 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260618 22:29:31.496965 30736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
I20260618 22:29:31.497231 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.496988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33526)
0618 22:29:31.497101 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.497155 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 22:29:31.497831 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.497657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33542)
0618 22:29:31.497740 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.497780 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260618 22:29:31.497958 30811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:46301: Network error: Client connection negotiation failed: client connection to 127.16.243.188:46301: connect: Connection refused (error 111)
W20260618 22:29:31.500442 30826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:31.500437 30823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:31.500562 30817 server_base.cc:1034] running on GCE node
W20260618 22:29:31.500437 30824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:31.501200 30817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:31.502341 30817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:31.503532 30817 hybrid_clock.cc:648] HybridClock initialized: now 1781821771503500 us; error 41 us; skew 500 ppm
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:31.506214 30817 init.cc:377] Logged in from keytab as kudu/127.16.243.188@KRBTEST.COM (short username kudu)
I20260618 22:29:31.507181 30817 webserver.cc:460] Webserver started at http://127.16.243.188:42659/ using document root <none> and password file <none>
I20260618 22:29:31.507746 30817 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:31.507817 30817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:31.508030 30817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:31.510244 30817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data/instance:
uuid: "9d2d77dc6a9f4ba98b653b907e024a9a"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "cc7a8f321eebaaf22e60394cd10fbc8d"
server_key_iv: "81fa729e9a717e88517e3f8e3709cb05"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.510843 30817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal/instance:
uuid: "9d2d77dc6a9f4ba98b653b907e024a9a"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "cc7a8f321eebaaf22e60394cd10fbc8d"
server_key_iv: "81fa729e9a717e88517e3f8e3709cb05"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.514652 30817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:29:31.517228 30833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:31.518625 30817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:31.518749 30817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "9d2d77dc6a9f4ba98b653b907e024a9a"
format_stamp: "Formatted at 2026-06-18 22:29:31 on dist-test-slave-jm5g"
server_key: "cc7a8f321eebaaf22e60394cd10fbc8d"
server_key_iv: "81fa729e9a717e88517e3f8e3709cb05"
server_key_version: "encryptionkey@0"
I20260618 22:29:31.518874 30817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:31.559289 30817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:31.562633 30817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:31.562844 30817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:31.569571 30817 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.188:46301
I20260618 22:29:31.569825 30885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.188:46301 every 8 connection(s)
I20260618 22:29:31.570576 30817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
I20260618 22:29:31.571192 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30817
I20260618 22:29:31.571316 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821616719733-17358-0/minicluster-data/master-2/wal/instance
I20260618 22:29:31.571570 17358 external_mini_cluster.cc:1428] Setting key e650a51834c180d8044a1366fb2596a7
I20260618 22:29:31.573657 30886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:31.577400 30886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:31.585088 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.572970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.16.243.188:46301)
0618 22:29:31.573398 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.573413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.573448 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:29:31.574122 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.574141 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.574490 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.574856 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.575623 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.576624 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.577452 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.577974 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.578120 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.580634 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.580639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.580648 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.580685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.582309 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.582770 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.582775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.582780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.582893 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.583147 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.583150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.583152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.583558 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.583758 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.584017 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.584294 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":138,"threads_started":1}
I20260618 22:29:31.588143 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.579056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42667 (local address 127.0.0.1:47288)
0618 22:29:31.579456 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.579505 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.579625 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.580408 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.580413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.580764 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.580886 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.580892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.581737 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.581740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.582380 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.582390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.582858 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.582862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.582971 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.583436 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.583447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.584758 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.586392 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.586400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.586405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.586664 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.586958 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.586962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.586964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.587045 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.587549 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.587556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.587679 (+ 123us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.587821 (+ 142us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.587994 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260618 22:29:31.588503 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.579155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.16.243.190:42667)
0618 22:29:31.579305 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.579310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.580097 (+ 787us) server_negotiation.cc:407] Connection header received
0618 22:29:31.580154 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.580157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.580212 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.580304 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.581008 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.581625 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.582492 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.582759 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.582848 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.584893 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.584897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.584899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.584925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.586291 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.586774 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.586778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.586780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.586874 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.587135 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.587139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.587141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.587390 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.587492 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.587942 (+ 450us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.588068 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:29:31.589110 30811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 3
I20260618 22:29:31.589465 30886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:29:31.596123 30736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } attempt: 3
I20260618 22:29:31.598095 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.587234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.16.243.190:42667)
0618 22:29:31.587612 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.587624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.587636 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:29:31.587667 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.587669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.587725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.587815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.588712 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.589293 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.590254 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.590652 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.590755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.593496 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.593501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.593504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.593531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.595587 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.596068 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.596073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.596075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.596550 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.597238 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.597242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.597244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.597460 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.597664 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.597795 (+ 131us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.597915 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260618 22:29:31.598956 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.589579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33568)
0618 22:29:31.589770 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.589785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.589881 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.590140 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.590144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.590334 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.590441 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.590449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.591445 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.591448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.592250 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.592258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.592860 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.592864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.592953 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.593433 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.593450 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.594875 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.596796 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.596800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.596802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.597043 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.597814 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.597817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.597819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.597909 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.598346 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.598349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.598452 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.598650 (+ 198us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.598785 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260618 22:29:31.599334 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.589938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:36997 (local address 127.0.0.1:52186)
0618 22:29:31.590141 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.590152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.590234 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.590987 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.590990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.591172 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.591281 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.591288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.592338 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.592343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.592949 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.592960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.593560 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.593563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.593634 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.594041 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.594055 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.595440 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.597714 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.597717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.597719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.597945 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.598254 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.598257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.598259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.598341 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.598808 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.598812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.598904 (+ 92us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.599017 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.599165 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2}
I20260618 22:29:31.599346 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.590637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.16.243.189:36997)
0618 22:29:31.590758 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.590762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.590775 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:29:31.590806 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.590809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.590870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.590946 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.591426 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.592220 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.593070 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.593375 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.593459 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.595604 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.595609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.595612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.595641 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.597439 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.598049 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.598052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.598054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.598136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.598438 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.598442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.598444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.598665 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.598731 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.599095 (+ 364us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.599205 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":45}
I20260618 22:29:31.599555 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.589617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.16.243.188:46301)
0618 22:29:31.589721 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.589724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.589836 (+ 112us) server_negotiation.cc:407] Connection header received
0618 22:29:31.589975 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.589978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.590021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.590092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.590588 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.591312 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.592375 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.592742 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.592848 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.595003 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.595007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.595010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.595036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.596663 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.597598 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.597601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.597603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.597704 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.598025 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.598028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.598030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.598238 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.599242 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.599316 (+ 74us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.599419 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":43}
I20260618 22:29:31.600168 30886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:29:31.603013 30811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: Bootstrap starting.
I20260618 22:29:31.605913 30811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: Neither blocks nor log segments found. Creating new log.
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:29:31 dist-test-slave-jm5g krb5kdc[30655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:29:31.608682 30811 log.cc:824] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:31.611466 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.600644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33580)
0618 22:29:31.600734 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.600746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.600812 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.601089 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.601092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.601235 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.601317 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.601323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.602243 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.602246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.602924 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.602934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.603352 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.603355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.603449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.603868 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.603885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.608015 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.609964 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.609967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.609969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.610161 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.610426 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.610432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.610577 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.611022 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.611025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.611098 (+ 73us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.611179 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.611310 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260618 22:29:31.611581 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.596510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:46301 (local address 127.0.0.1:33574)
0618 22:29:31.596773 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:31.596786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:31.596870 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:31.597309 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:31.597313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:31.597484 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:31.597594 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.597602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.600783 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.600788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.601465 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:31.601477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.602175 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.602179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:31.602266 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.602838 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:31.602852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:31.607480 (+ 4628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:31.609853 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.609857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.609859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.610121 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.610468 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:31.610474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:31.610617 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:31.611102 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:31.611106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:31.611209 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:29:31.611287 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:29:31.611420 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260618 22:29:31.611626 30811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: No bootstrap required, opened a new log
I20260618 22:29:31.611908 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.600786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.16.243.188:46301)
0618 22:29:31.600879 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.600884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.600896 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:29:31.600928 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.600931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.600972 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.601047 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.601424 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.602140 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.603033 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.603257 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.603347 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.608184 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.608188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.608191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.608215 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.609873 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610247 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.610250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.610252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.610336 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610684 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.610686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.610688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.610901 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.610990 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.611677 (+ 687us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.611780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260618 22:29:31.613009 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.599500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.16.243.189:36997)
0618 22:29:31.599625 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.599629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.600416 (+ 787us) server_negotiation.cc:407] Connection header received
0618 22:29:31.603566 (+ 3150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.603569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.603615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.603674 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.604468 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.604974 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.605949 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.606177 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.606267 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.609652 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.609655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.609658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.609692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.611184 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.611647 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.611650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.611652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.611736 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.612046 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.612050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.612051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.612273 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.612409 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.612756 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.612871 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:29:31.614923 30886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: Bootstrap starting.
I20260618 22:29:31.615190 30811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.615355 30811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:31.615387 30811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21366e1a98024de8a61a98b9320b0eb8, State: Initialized, Role: FOLLOWER
I20260618 22:29:31.615768 30811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.616180 30894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.616297 30894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.617004 30811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:31.617098 30886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:31.617926 30886 log.cc:824] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:31.620842 30886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: No bootstrap required, opened a new log
I20260618 22:29:31.621237 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:29:31.623046 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.596748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.16.243.188:46301)
0618 22:29:31.597091 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.597097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.597107 (+ 10us) server_negotiation.cc:407] Connection header received
0618 22:29:31.597132 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.597134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.597166 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.597280 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.599474 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.600645 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.601618 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.602048 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.602159 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.607679 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.607684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.607687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.607716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.609715 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610247 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.610250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.610252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.610364 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.610742 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.610746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.610748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.610983 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.622670 (+ 11687us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.622780 (+ 110us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.622907 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":59520,"thread_start_us":127,"threads_started":1}
I20260618 22:29:31.621389 30743 master_runner.cc:186] Error getting master registration for 127.16.243.190:42667: OK, instance_id { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" instance_seqno: 1781821771133585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:29:31.625310 30743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:29:31.625658 30886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.625921 30886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:31.625968 30886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2d77dc6a9f4ba98b653b907e024a9a, State: Initialized, Role: FOLLOWER
I20260618 22:29:31.626148 30736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: Bootstrap starting.
I20260618 22:29:31.626462 30886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.627821 30886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:31.628993 30736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: Neither blocks nor log segments found. Creating new log.
W20260618 22:29:31.633399 30925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:29:31.633478 30925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:29:31.634107 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.623562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.16.243.190:42667)
0618 22:29:31.623693 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.623697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.624821 (+ 1124us) server_negotiation.cc:407] Connection header received
0618 22:29:31.625252 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.625255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.625302 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.625351 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.626330 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.627138 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.628423 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.628797 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.628869 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.630543 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.630583 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.630586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.630613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.632229 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.632699 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.632703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.632705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.632797 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.633111 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.633114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.633116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.633338 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.633457 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.633824 (+ 367us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.633946 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:29:31.631284 30817 master_runner.cc:186] Error getting master registration for 127.16.243.190:42667: OK, instance_id { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" instance_seqno: 1781821771133585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:29:31.635293 30817 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:29:31.635435 30736 log.cc:824] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:31.630127 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.635990 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.636322 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.625481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.16.243.188:46301)
0618 22:29:31.626279 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.626284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.626297 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:29:31.626331 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.626333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.626376 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.626438 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.627349 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.628200 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.629229 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.629866 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.629966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.632020 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.632024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.632027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.632053 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.633783 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.634505 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.634508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.634510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.634616 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.634886 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.634889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.634891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.635087 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.635239 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.635951 (+ 712us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.636178 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662}
W20260618 22:29:31.636909 30906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:29:31.636982 30906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:29:31.637890 30736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: No bootstrap required, opened a new log
I20260618 22:29:31.641837 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:31.624239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.16.243.189:36997)
0618 22:29:31.624343 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:31.624347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:31.634723 (+ 10376us) server_negotiation.cc:407] Connection header received
0618 22:29:31.634896 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:31.634899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:31.634949 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:31.635000 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:31.635735 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.636434 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.637405 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:31.637646 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:31.637712 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:31.639104 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:31.639108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:31.639111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:31.639136 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:31.640377 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.640742 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.640746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.640747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.640819 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:31.641048 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:31.641051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:31.641053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:31.641233 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:31.641362 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:31.641570 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:29:31.641677 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:29:31.642011 30736 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.642266 30736 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:31.642320 30736 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff57730309c460eaec54afb77e48d4f, State: Initialized, Role: FOLLOWER
I20260618 22:29:31.642880 30736 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.643419 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.643546 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.644407 30736 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:29:31.650741 30940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:29:31.650839 30940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:29:31.656546 30668 master_runner.cc:263] No leader master found from master cff57730309c460eaec54afb77e48d4f
I20260618 22:29:31.656706 30668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:29:31.695778 30914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:31.695987 30914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.697096 30914 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff57730309c460eaec54afb77e48d4f (127.16.243.190:42667), 21366e1a98024de8a61a98b9320b0eb8 (127.16.243.189:36997)
I20260618 22:29:31.697916 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21366e1a98024de8a61a98b9320b0eb8" is_pre_election: true
I20260618 22:29:31.698261 30785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d77dc6a9f4ba98b653b907e024a9a in term 0.
I20260618 22:29:31.698737 30837 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21366e1a98024de8a61a98b9320b0eb8, 9d2d77dc6a9f4ba98b653b907e024a9a; no voters:
I20260618 22:29:31.699455 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff57730309c460eaec54afb77e48d4f" is_pre_election: true
I20260618 22:29:31.699757 30710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d77dc6a9f4ba98b653b907e024a9a in term 0.
I20260618 22:29:31.700338 30914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:31.700433 30914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:31.700482 30914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:31.701807 30914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.702507 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff57730309c460eaec54afb77e48d4f"
I20260618 22:29:31.702658 30710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:31.704493 30710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d77dc6a9f4ba98b653b907e024a9a in term 1.
I20260618 22:29:31.704835 30836 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2d77dc6a9f4ba98b653b907e024a9a, cff57730309c460eaec54afb77e48d4f; no voters:
I20260618 22:29:31.708104 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21366e1a98024de8a61a98b9320b0eb8"
I20260618 22:29:31.708259 30785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:31.709631 30785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d77dc6a9f4ba98b653b907e024a9a in term 1.
I20260618 22:29:31.707716 30914 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [CANDIDATE]: Term 1 election: Requested vote from peers cff57730309c460eaec54afb77e48d4f (127.16.243.190:42667), 21366e1a98024de8a61a98b9320b0eb8 (127.16.243.189:36997)
I20260618 22:29:31.710168 30914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:31.710274 30914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [term 1 LEADER]: Becoming Leader. State: Replica: 9d2d77dc6a9f4ba98b653b907e024a9a, State: Running, Role: LEADER
I20260618 22:29:31.710687 30914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } }
I20260618 22:29:31.712560 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2d77dc6a9f4ba98b653b907e024a9a. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.712670 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:31.713030 30944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:31.714123 30944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:31.724301 30785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [term 1 FOLLOWER]: Refusing update from remote peer 9d2d77dc6a9f4ba98b653b907e024a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:31.725000 30942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:31.729244 30894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2d77dc6a9f4ba98b653b907e024a9a. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.729378 30894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.733917 30710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [term 1 FOLLOWER]: Refusing update from remote peer 9d2d77dc6a9f4ba98b653b907e024a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:31.734504 30942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:31.738770 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2d77dc6a9f4ba98b653b907e024a9a. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.738899 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.746984 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.747146 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:31.748266 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.748390 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.748762 30945 mvcc.cc:205] Tried to move back new op lower bound from 7298341976974041088 to 7298341976930349056. Current Snapshot: MvccSnapshot[applied={T|T < 7298341976974041088}]
I20260618 22:29:31.751466 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.751595 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:31.752709 30894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff57730309c460eaec54afb77e48d4f" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42667 } } peers { permanent_uuid: "21366e1a98024de8a61a98b9320b0eb8" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 36997 } } peers { permanent_uuid: "9d2d77dc6a9f4ba98b653b907e024a9a" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 46301 } } }
I20260618 22:29:31.752838 30894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:29:31.753867 30948 mvcc.cc:205] Tried to move back new op lower bound from 7298341976974041088 to 7298341976930349056. Current Snapshot: MvccSnapshot[applied={T|T < 7298341976974041088}]
I20260618 22:29:31.754040 30946 mvcc.cc:205] Tried to move back new op lower bound from 7298341976974041088 to 7298341976930349056. Current Snapshot: MvccSnapshot[applied={T|T < 7298341976974041088}]
I20260618 22:29:31.760550 30944 catalog_manager.cc:1347] Generated new cluster ID: 9ea8101d366945adbbd5b684aebb646d
I20260618 22:29:31.760638 30944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:31.802701 30944 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:29:31.804008 30944 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:31.820067 30944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2d77dc6a9f4ba98b653b907e024a9a: Generated new TSK 0
I20260618 22:29:31.821219 30944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:29:32.638262 30906 catalog_manager.cc:1259] Loaded cluster ID: 9ea8101d366945adbbd5b684aebb646d
I20260618 22:29:32.638342 30906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: loading cluster ID for follower catalog manager: success
I20260618 22:29:32.639997 30906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: acquiring CA information for follower catalog manager: success
I20260618 22:29:32.641122 30906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21366e1a98024de8a61a98b9320b0eb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:29:32.652153 30940 catalog_manager.cc:1259] Loaded cluster ID: 9ea8101d366945adbbd5b684aebb646d
I20260618 22:29:32.652246 30940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: loading cluster ID for follower catalog manager: success
I20260618 22:29:32.653949 30940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: acquiring CA information for follower catalog manager: success
I20260618 22:29:32.655184 30940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cff57730309c460eaec54afb77e48d4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:29:32.882743 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30668
I20260618 22:29:32.887876 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30743
I20260618 22:29:32.892546 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30817
2026-06-18T22:29:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3015 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:29:32 dist-test-slave-jm5g krb5kdc[30958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:29:32 dist-test-slave-jm5g krb5kdc[30958](info): set up 2 sockets
Jun 18 22:29:32 dist-test-slave-jm5g krb5kdc[30958](info): commencing operation
krb5kdc: starting...
W20260618 22:29:33.850505 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:29:33 dist-test-slave-jm5g krb5kdc[30958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:29:33.965478 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:39693
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:39693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:34.073757 30971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:34.074052 30971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:34.074110 30971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:34.078188 30971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:34.078266 30971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:34.078301 30971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:34.078323 30971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:34.078351 30971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:34.083894 30971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:39693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:39693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:34.085021 30971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:34.086066 30971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:34.094485 30979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:34.094501 30977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:34.094592 30971 server_base.cc:1034] running on GCE node
W20260618 22:29:34.094485 30976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:34.095465 30971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:34.096644 30971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:34.097816 30971 hybrid_clock.cc:648] HybridClock initialized: now 1781821774097800 us; error 35 us; skew 500 ppm
Jun 18 22:29:34 dist-test-slave-jm5g krb5kdc[30958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:34.100651 30971 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:34.101645 30971 webserver.cc:460] Webserver started at http://127.16.243.190:39091/ using document root <none> and password file <none>
I20260618 22:29:34.102217 30971 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:34.102278 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:34.102485 30971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:34.104679 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "a9a7635f284448f8939f76cf49d57b17"
format_stamp: "Formatted at 2026-06-18 22:29:34 on dist-test-slave-jm5g"
server_key: "a99ab93b1a899a87b7010c0ca335f565"
server_key_iv: "baa81e4c455a6f92dd1755d3ff13f72d"
server_key_version: "encryptionkey@0"
I20260618 22:29:34.105259 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "a9a7635f284448f8939f76cf49d57b17"
format_stamp: "Formatted at 2026-06-18 22:29:34 on dist-test-slave-jm5g"
server_key: "a99ab93b1a899a87b7010c0ca335f565"
server_key_iv: "baa81e4c455a6f92dd1755d3ff13f72d"
server_key_version: "encryptionkey@0"
I20260618 22:29:34.109189 30971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:29:34.111827 30986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:34.113160 30971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 22:29:34.113286 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "a9a7635f284448f8939f76cf49d57b17"
format_stamp: "Formatted at 2026-06-18 22:29:34 on dist-test-slave-jm5g"
server_key: "a99ab93b1a899a87b7010c0ca335f565"
server_key_iv: "baa81e4c455a6f92dd1755d3ff13f72d"
server_key_version: "encryptionkey@0"
I20260618 22:29:34.113404 30971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:34.130975 30971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:34.134370 30971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:34.134603 30971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:34.141650 30971 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:39693
I20260618 22:29:34.141675 31038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:39693 every 8 connection(s)
I20260618 22:29:34.142706 30971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:34.145527 31039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:34.151206 31039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: Bootstrap starting.
I20260618 22:29:34.151818 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 30971
I20260618 22:29:34.151923 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:29:34.152175 17358 external_mini_cluster.cc:1428] Setting key 83b0931130a3b0ad9d2b2626891fdf4f
I20260618 22:29:34.153929 31039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:34.154863 31039 log.cc:824] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:34.157186 31039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: No bootstrap required, opened a new log
Jun 18 22:29:34 dist-test-slave-jm5g krb5kdc[30958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:34.161418 31039 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } }
I20260618 22:29:34.161722 31039 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:34.161821 31039 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a7635f284448f8939f76cf49d57b17, State: Initialized, Role: FOLLOWER
I20260618 22:29:34.162324 31039 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } }
I20260618 22:29:34.162494 31039 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:34.162586 31039 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:34.162695 31039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:34.163933 31039 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } }
I20260618 22:29:34.164338 31039 leader_election.cc:302] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a7635f284448f8939f76cf49d57b17; no voters:
I20260618 22:29:34.164690 31039 leader_election.cc:288] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:29:34.164839 31044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:34.165053 31044 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [term 1 LEADER]: Becoming Leader. State: Replica: a9a7635f284448f8939f76cf49d57b17, State: Running, Role: LEADER
I20260618 22:29:34.165469 31044 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } }
I20260618 22:29:34.165956 31039 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:34.167268 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a7635f284448f8939f76cf49d57b17. Latest consensus state: current_term: 1 leader_uuid: "a9a7635f284448f8939f76cf49d57b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } } }
I20260618 22:29:34.167413 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:34.167768 31045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a7635f284448f8939f76cf49d57b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a7635f284448f8939f76cf49d57b17" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 39693 } } }
I20260618 22:29:34.167918 31045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:34.168073 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:34.153477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.16.243.190:39693)
0618 22:29:34.153940 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:34.153960 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:34.154005 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:29:34.154939 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:34.154965 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:34.155402 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:34.155721 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:34.156431 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:34.157870 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:34.158771 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:34.159497 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:34.159669 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:34.162304 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:34.162310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:34.162328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:34.162364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:34.164485 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:34.165105 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:34.165112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:34.165118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:34.165276 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:34.165612 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:34.165615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:34.165617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:34.166148 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:34.166337 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:34.166657 (+ 320us) server_negotiation.cc:299] Negotiation successful
0618 22:29:34.167003 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
W20260618 22:29:34.170958 31059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:29:34.171029 31059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:29:34.171087 31053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:34.171617 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:29:34.171833 31053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:34.177017 31053 catalog_manager.cc:1347] Generated new cluster ID: 6a54defa501041f3afeff17319ff5f7d
I20260618 22:29:34.177094 31053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:34.184592 31053 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:29:34.185933 31053 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:34.194024 31053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a7635f284448f8939f76cf49d57b17: Generated new TSK 0
I20260618 22:29:34.194944 31053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:29:34.227521 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 30971
2026-06-18T22:29:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1333 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:29:34 dist-test-slave-jm5g krb5kdc[31066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:29:34 dist-test-slave-jm5g krb5kdc[31066](info): set up 2 sockets
Jun 18 22:29:34 dist-test-slave-jm5g krb5kdc[31066](info): commencing operation
krb5kdc: starting...
W20260618 22:29:35.174788 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:29:35.287058 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:35.394289 31079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:35.394613 31079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:35.394671 31079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:35.398841 31079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:35.398911 31079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:35.398929 31079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:35.398945 31079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:35.398959 31079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:35.404294 31079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:35.405330 31079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:35.406353 31079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:35.414404 31084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:35.414417 31085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:35.414489 31087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:35.414714 31079 server_base.cc:1034] running on GCE node
I20260618 22:29:35.415329 31079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:35.416406 31079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:35.417552 31079 hybrid_clock.cc:648] HybridClock initialized: now 1781821775417535 us; error 28 us; skew 500 ppm
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:35.431146 31079 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:35.432260 31079 webserver.cc:460] Webserver started at http://127.16.243.190:35803/ using document root <none> and password file <none>
I20260618 22:29:35.432837 31079 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:35.432898 31079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:35.433066 31079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:35.435209 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "84007392c5bb438384faab0c6ba8aa18"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "1f74fc5c40f624b6a8916348f6531a3e"
server_key_iv: "84d16c73ccb884681a1783c1b0124848"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.435737 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "84007392c5bb438384faab0c6ba8aa18"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "1f74fc5c40f624b6a8916348f6531a3e"
server_key_iv: "84d16c73ccb884681a1783c1b0124848"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.439400 31079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:35.441742 31094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.443070 31079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:29:35.443176 31079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "84007392c5bb438384faab0c6ba8aa18"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "1f74fc5c40f624b6a8916348f6531a3e"
server_key_iv: "84d16c73ccb884681a1783c1b0124848"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.443269 31079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:35.475231 31079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:35.478204 31079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:35.478363 31079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:35.485110 31079 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41667
I20260618 22:29:35.485369 31146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41667 every 8 connection(s)
I20260618 22:29:35.486080 31079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:35.488754 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:35.493337 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31079
I20260618 22:29:35.493451 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:29:35.493707 17358 external_mini_cluster.cc:1428] Setting key 355ed6766adc0e9c82bb4962dc793014
I20260618 22:29:35.494304 31147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap starting.
I20260618 22:29:35.496595 31147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:35.497485 31147 log.cc:824] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:35.499577 31147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: No bootstrap required, opened a new log
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:35.503862 31147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:35.504104 31147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:35.504210 31147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Initialized, Role: FOLLOWER
I20260618 22:29:35.504769 31147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:35.504932 31147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:35.505000 31147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:35.505100 31147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:35.506165 31147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:35.506573 31147 leader_election.cc:302] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84007392c5bb438384faab0c6ba8aa18; no voters:
I20260618 22:29:35.506935 31147 leader_election.cc:288] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:29:35.507052 31152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:35.507463 31152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 LEADER]: Becoming Leader. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Running, Role: LEADER
I20260618 22:29:35.507838 31152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:35.508278 31147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:35.508565 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:35.495020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.16.243.190:41667)
0618 22:29:35.495438 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:35.495453 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:35.495487 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:29:35.496121 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:35.496139 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:35.496471 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:35.496800 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:35.497661 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.498715 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.499840 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.500322 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.500441 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:35.503203 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:35.503208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:35.503225 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:35.503256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:35.505145 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:35.505590 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:35.505597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:35.505603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:35.505749 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:35.506079 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:35.506082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:35.506084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:35.506683 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:35.506890 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:35.507140 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 22:29:35.507457 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260618 22:29:35.510334 31154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:35.510380 31153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84007392c5bb438384faab0c6ba8aa18. Latest consensus state: current_term: 1 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:35.510507 31154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:35.510543 31153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
W20260618 22:29:35.513564 31166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:29:35.513664 31166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:29:35.513716 31167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:35.514514 31167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:35.520180 31167 catalog_manager.cc:1347] Generated new cluster ID: c88ac29d8552487c881d25c66db71655
I20260618 22:29:35.520248 31167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:35.533685 31167 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:29:35.534727 31167 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:35.555370 31167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Generated new TSK 0
I20260618 22:29:35.556110 31167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:29:35.583338 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:35.699681 31173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:35.699980 31173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:35.700024 31173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:35.704156 31173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:35.704236 31173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:35.704298 31173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:35.709831 31173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:35.711035 31173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:35.712055 31173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:35.721743 31178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:35.721735 31179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:35.721787 31173 server_base.cc:1034] running on GCE node
W20260618 22:29:35.721789 31181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:35.722298 31173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:35.723069 31173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:35.724258 31173 hybrid_clock.cc:648] HybridClock initialized: now 1781821775724217 us; error 55 us; skew 500 ppm
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:35.727186 31173 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:35.728159 31173 webserver.cc:460] Webserver started at http://127.16.243.129:33619/ using document root <none> and password file <none>
I20260618 22:29:35.728734 31173 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:35.728806 31173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:35.729012 31173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:35.731246 31173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "d029dcc7b338446dae794046815546ac"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "3d95f8e73e3b29f76844a39160295944"
server_key_iv: "797d62df73eb8f02b089603415623f60"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.731809 31173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "d029dcc7b338446dae794046815546ac"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "3d95f8e73e3b29f76844a39160295944"
server_key_iv: "797d62df73eb8f02b089603415623f60"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.735531 31173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:35.738082 31188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.739382 31173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:35.739503 31173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "d029dcc7b338446dae794046815546ac"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "3d95f8e73e3b29f76844a39160295944"
server_key_iv: "797d62df73eb8f02b089603415623f60"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.739600 31173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:35.763875 31173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:35.767011 31173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:35.767179 31173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:35.767748 31173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:35.768541 31173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:35.768585 31173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.768623 31173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:35.768643 31173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.777242 31173 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:33007
I20260618 22:29:35.777261 31301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:33007 every 8 connection(s)
I20260618 22:29:35.778147 31173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:35.780283 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31173
I20260618 22:29:35.780418 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:29:35.780738 17358 external_mini_cluster.cc:1428] Setting key 17bfd2cd141103dd426e89bb4a03736e
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:29:35.792937 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:35.780025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:55667 (local address 127.16.243.190:41667)
0618 22:29:35.780156 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:35.780159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:35.781238 (+ 1079us) server_negotiation.cc:407] Connection header received
0618 22:29:35.782725 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:35.782728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:35.782785 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:35.782874 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:35.784269 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.784796 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.786417 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.786718 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.786841 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:35.789463 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:35.789468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:35.789471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:35.789503 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:35.790964 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:35.791445 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:35.791448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:35.791449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:35.791517 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:35.791852 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:35.791854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:35.791855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:35.792045 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:35.792117 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:35.792668 (+ 551us) server_negotiation.cc:299] Negotiation successful
0618 22:29:35.792805 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:29:35.794046 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:35.780472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:55667)
0618 22:29:35.781057 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:35.781107 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:35.782485 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:35.783033 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:35.783040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:35.783426 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:35.784119 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:35.784137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.784920 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.784923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:35.786286 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:35.786298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:35.786879 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:35.786883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:35.787031 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:35.787630 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:35.787647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:35.789282 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:35.791065 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:35.791070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:35.791084 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:35.791363 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:35.791578 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:35.791580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:35.791581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:35.791747 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:35.792131 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:35.792136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:35.792431 (+ 295us) client_negotiation.cc:769] Sending connection context
0618 22:29:35.792613 (+ 182us) client_negotiation.cc:240] Negotiation successful
0618 22:29:35.792976 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":144,"threads_started":1}
I20260618 22:29:35.795352 31302 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:35.795615 31302 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:35.796198 31302 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
I20260618 22:29:35.797842 31111 ts_manager.cc:194] Registered new tserver with Master: d029dcc7b338446dae794046815546ac (127.16.243.129:33007)
I20260618 22:29:35.799046 31111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:55667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:29:35.807924 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:35.918335 31307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:35.918653 31307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:35.918735 31307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:35.922803 31307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:35.922866 31307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:35.922945 31307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:35.928354 31307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:35.929452 31307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:35.930476 31307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:35.939743 31313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:35.939772 31315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:35.939771 31312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:35.939944 31307 server_base.cc:1034] running on GCE node
I20260618 22:29:35.940424 31307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:35.941064 31307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:35.942212 31307 hybrid_clock.cc:648] HybridClock initialized: now 1781821775942183 us; error 42 us; skew 500 ppm
Jun 18 22:29:35 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:35.944998 31307 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:35.945854 31307 webserver.cc:460] Webserver started at http://127.16.243.130:35347/ using document root <none> and password file <none>
I20260618 22:29:35.946375 31307 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:35.946470 31307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:35.946699 31307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:35.948753 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "075d573206ce4462a14a28ac3e8a4f95"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "0e6ecc5503402735f21d67b87f64bffd"
server_key_iv: "e3e940afab602e7cd7fb3f8802b26c3e"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.949275 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "075d573206ce4462a14a28ac3e8a4f95"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "0e6ecc5503402735f21d67b87f64bffd"
server_key_iv: "e3e940afab602e7cd7fb3f8802b26c3e"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.953045 31307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:29:35.955585 31322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.956823 31307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:35.956930 31307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "075d573206ce4462a14a28ac3e8a4f95"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "0e6ecc5503402735f21d67b87f64bffd"
server_key_iv: "e3e940afab602e7cd7fb3f8802b26c3e"
server_key_version: "encryptionkey@0"
I20260618 22:29:35.957044 31307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:35.982097 31307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:35.985387 31307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:35.985569 31307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:35.986186 31307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:35.987079 31307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:35.987126 31307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.987181 31307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:35.987210 31307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:35.995682 31307 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36959
I20260618 22:29:35.995705 31435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36959 every 8 connection(s)
I20260618 22:29:35.996588 31307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:36.004443 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31307
I20260618 22:29:36.004575 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:29:36.004920 17358 external_mini_cluster.cc:1428] Setting key 2444e67f296a0d1fd8374d92554e95d7
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:36.011109 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:35.998363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57173 (local address 127.16.243.190:41667)
0618 22:29:35.998491 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:35.998494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:35.999442 (+ 948us) server_negotiation.cc:407] Connection header received
0618 22:29:36.000830 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.000834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.000893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.000963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:36.002174 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.002741 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.003886 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.004105 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.004192 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.006743 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:36.006746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:36.006748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:36.006778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:36.008386 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.009007 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.009009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.009011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.009114 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.009722 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.009726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.009727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.009972 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:36.010059 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.010797 (+ 738us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.010940 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:29:36.012055 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:35.998829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:57173)
0618 22:29:35.999286 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:35.999329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.000590 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.001154 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.001161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.001538 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:36.002029 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.002042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.002866 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.002869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.003769 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.003778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.004247 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.004251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.004419 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.005017 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:36.005038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:36.006578 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:36.008511 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:36.008518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:36.008537 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:36.008912 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:36.009474 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:36.009476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:36.009478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:36.009636 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:36.010104 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:36.010112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:36.010461 (+ 349us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.010709 (+ 248us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.011077 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260618 22:29:36.013195 31436 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:36.013465 31436 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:36.014083 31436 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:29:36.015244 31111 ts_manager.cc:194] Registered new tserver with Master: 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
I20260618 22:29:36.015903 31111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:57173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:29:36.030058 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:36.140713 31441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:36.140985 31441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:36.141028 31441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:36.145207 31441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:36.145282 31441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:36.145344 31441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:36.151278 31441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:36.152307 31441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:36.153288 31441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:36.162822 31447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:36.162827 31449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:36.163062 31441 server_base.cc:1034] running on GCE node
W20260618 22:29:36.162868 31446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:36.164191 31441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:36.164878 31441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:36.166064 31441 hybrid_clock.cc:648] HybridClock initialized: now 1781821776166043 us; error 36 us; skew 500 ppm
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:36.168797 31441 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:36.169667 31441 webserver.cc:460] Webserver started at http://127.16.243.131:46427/ using document root <none> and password file <none>
I20260618 22:29:36.170195 31441 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:36.170257 31441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:36.170439 31441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:36.172577 31441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "dc07fa4a091a4ab59aaea4eac6509266"
format_stamp: "Formatted at 2026-06-18 22:29:36 on dist-test-slave-jm5g"
server_key: "4dee38837dfaa5cdccd76ca7d506a5fc"
server_key_iv: "d07de0da53bc472893ac90afd76e0834"
server_key_version: "encryptionkey@0"
I20260618 22:29:36.173105 31441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "dc07fa4a091a4ab59aaea4eac6509266"
format_stamp: "Formatted at 2026-06-18 22:29:36 on dist-test-slave-jm5g"
server_key: "4dee38837dfaa5cdccd76ca7d506a5fc"
server_key_iv: "d07de0da53bc472893ac90afd76e0834"
server_key_version: "encryptionkey@0"
I20260618 22:29:36.176867 31441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:29:36.179260 31456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:36.180629 31441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:29:36.180754 31441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "dc07fa4a091a4ab59aaea4eac6509266"
format_stamp: "Formatted at 2026-06-18 22:29:36 on dist-test-slave-jm5g"
server_key: "4dee38837dfaa5cdccd76ca7d506a5fc"
server_key_iv: "d07de0da53bc472893ac90afd76e0834"
server_key_version: "encryptionkey@0"
I20260618 22:29:36.180874 31441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:36.204622 31441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:36.208055 31441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:36.208243 31441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:36.208895 31441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:36.209764 31441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:36.209812 31441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:36.209863 31441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:36.209887 31441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:36.218511 31441 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35905
I20260618 22:29:36.218539 31569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35905 every 8 connection(s)
I20260618 22:29:36.219483 31441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:36.226596 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31441
I20260618 22:29:36.226713 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:29:36.226967 17358 external_mini_cluster.cc:1428] Setting key 67c412a957d08fe7e6fd468dff2c8fd6
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:36.233342 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.221366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50221 (local address 127.16.243.190:41667)
0618 22:29:36.221546 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.221550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.222391 (+ 841us) server_negotiation.cc:407] Connection header received
0618 22:29:36.223784 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.223788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.223843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.223917 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:36.225119 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.225658 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.226917 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.227171 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.227262 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.229994 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:36.229997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:36.230000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:36.230024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:36.231354 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.231872 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.231875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.231876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.231952 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.232278 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.232281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.232283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.232466 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:36.232548 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.233053 (+ 505us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.233180 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260618 22:29:36.234229 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.221729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:50221)
0618 22:29:36.222233 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.222281 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.223567 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.224102 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.224109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.224493 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:36.224980 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.224993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.225778 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.225781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.226806 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.226816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.227318 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.227321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.227503 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.228192 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:36.228213 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:36.229832 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:36.231473 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:36.231480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:36.231500 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:36.231775 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:36.232032 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:36.232035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:36.232037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:36.232193 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:36.232561 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:36.232568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:36.232826 (+ 258us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.232999 (+ 173us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.233354 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":116,"threads_started":1}
I20260618 22:29:36.235421 31570 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:36.235674 31570 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:36.236184 31570 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
I20260618 22:29:36.237241 31111 ts_manager.cc:194] Registered new tserver with Master: dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:36.237819 31111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:50221
I20260618 22:29:36.241317 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:29:36.250154 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.243225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.16.243.190:41667)
0618 22:29:36.243365 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.243370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.243533 (+ 163us) server_negotiation.cc:407] Connection header received
0618 22:29:36.243656 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.243659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.243705 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.243770 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:36.244496 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.245018 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.245845 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.246059 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.246144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.247541 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:36.247544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:36.247547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:36.247572 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:36.248702 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.249093 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.249097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.249099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.249193 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:36.249444 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:36.249446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:36.249448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:36.249672 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:36.249791 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.249925 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.250015 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:29:36.252872 31111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:29:36.255856 31111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:29:36.270071 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.264039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33007 (local address 127.0.0.1:41968)
0618 22:29:36.265233 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.265245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.265325 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.265809 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.265814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.265831 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.266033 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.266041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267845 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.267848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.268669 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.268678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.269736 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.269740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.269820 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.269839 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.269889 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.269939 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":10,"thread_start_us":83,"threads_started":1}
I20260618 22:29:36.270171 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.264827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.16.243.129:33007)
0618 22:29:36.265213 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.265242 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.265424 (+ 182us) server_negotiation.cc:407] Connection header received
0618 22:29:36.265498 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.265502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.265672 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.265815 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.266682 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267732 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.268783 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.269617 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.269752 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.269884 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.269933 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.270026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260618 22:29:36.270426 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.264839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.16.243.131:35905)
0618 22:29:36.266041 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.266050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.266066 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:29:36.266130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.266134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.266314 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.266463 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.266715 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267520 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.268806 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.269381 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.269494 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.269638 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.270134 (+ 496us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.270295 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":130,"threads_started":1}
I20260618 22:29:36.270071 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.264344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.0.0.1:60234)
0618 22:29:36.265366 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.265379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.265455 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.266472 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.266474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.266491 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.266598 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.266604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267668 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.267671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.268691 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.268700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.269499 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.269502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.269606 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.269656 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.269765 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.269895 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260618 22:29:36.271307 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.263999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.0.0.1:40432)
0618 22:29:36.264841 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.264893 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.265090 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.265833 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.265836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.265853 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.266033 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.266041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267486 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.267489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.269696 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.269706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.270916 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.270920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.271008 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.271027 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.271092 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.271174 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":1068,"thread_start_us":91,"threads_started":1}
I20260618 22:29:36.271529 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.264126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.16.243.130:36959)
0618 22:29:36.264571 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.264582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.264982 (+ 400us) server_negotiation.cc:407] Connection header received
0618 22:29:36.265273 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.265279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.265498 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.265635 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.266162 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.267340 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.269822 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.270627 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.270745 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.270858 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.271147 (+ 289us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.271239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":124,"threads_started":1}
I20260618 22:29:36.273702 31504 tablet_service.cc:1467] Processing CreateTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a (DEFAULT_TABLE table=test-table [id=5b32f4c0e68b41bb900868fafcb55a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:36.273702 31236 tablet_service.cc:1467] Processing CreateTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a (DEFAULT_TABLE table=test-table [id=5b32f4c0e68b41bb900868fafcb55a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:36.273897 31370 tablet_service.cc:1467] Processing CreateTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a (DEFAULT_TABLE table=test-table [id=5b32f4c0e68b41bb900868fafcb55a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:36.274860 31370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7ea3bfde854c06a9dea42c7ba4231a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:36.274870 31504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7ea3bfde854c06a9dea42c7ba4231a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:36.275039 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7ea3bfde854c06a9dea42c7ba4231a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:36.281061 31587 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Bootstrap starting.
I20260618 22:29:36.282603 31588 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Bootstrap starting.
I20260618 22:29:36.282603 31589 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Bootstrap starting.
I20260618 22:29:36.283766 31587 tablet_bootstrap.cc:654] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:36.284526 31587 log.cc:824] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:36.284842 31588 tablet_bootstrap.cc:654] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:36.285351 31589 tablet_bootstrap.cc:654] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:36.286082 31589 log.cc:824] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:36.286080 31588 log.cc:824] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:36.286427 31587 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: No bootstrap required, opened a new log
I20260618 22:29:36.286656 31587 ts_tablet_manager.cc:1397] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:29:36.287773 31588 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: No bootstrap required, opened a new log
I20260618 22:29:36.287940 31588 ts_tablet_manager.cc:1397] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:29:36.288089 31589 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: No bootstrap required, opened a new log
I20260618 22:29:36.288266 31589 ts_tablet_manager.cc:1397] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:29:36.290583 31587 raft_consensus.cc:348] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.290885 31587 raft_consensus.cc:374] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:36.290940 31587 raft_consensus.cc:729] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075d573206ce4462a14a28ac3e8a4f95, State: Initialized, Role: FOLLOWER
I20260618 22:29:36.291028 31588 raft_consensus.cc:348] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.291272 31588 raft_consensus.cc:374] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:36.291323 31588 raft_consensus.cc:729] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d029dcc7b338446dae794046815546ac, State: Initialized, Role: FOLLOWER
I20260618 22:29:36.291404 31587 consensus_queue.cc:260] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.291713 31588 consensus_queue.cc:260] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.291824 31589 raft_consensus.cc:348] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.292048 31589 raft_consensus.cc:374] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:36.292099 31589 raft_consensus.cc:729] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc07fa4a091a4ab59aaea4eac6509266, State: Initialized, Role: FOLLOWER
I20260618 22:29:36.292239 31436 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
I20260618 22:29:36.292495 31587 ts_tablet_manager.cc:1428] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:29:36.292522 31589 consensus_queue.cc:260] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.292639 31588 ts_tablet_manager.cc:1428] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:29:36.292941 31302 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
I20260618 22:29:36.293114 31570 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
I20260618 22:29:36.293383 31589 ts_tablet_manager.cc:1428] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260618 22:29:36.470582 31571 tablet.cc:2369] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:29:36.497722 31437 tablet.cc:2369] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:36.516134 31594 raft_consensus.cc:482] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:36.516364 31594 raft_consensus.cc:504] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.517517 31594 leader_election.cc:288] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959), dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:36.518190 31593 raft_consensus.cc:482] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:36.518366 31593 raft_consensus.cc:504] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.519503 31593 leader_election.cc:288] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d029dcc7b338446dae794046815546ac (127.16.243.129:33007), dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:36.522662 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.517806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.16.243.129:34021)
0618 22:29:36.518176 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.518193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.518298 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.518659 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.518661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.518694 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.518786 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.518790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.519903 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.519906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.521292 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.521302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.522202 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.522299 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.522324 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.522384 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.522453 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":64,"spinlock_wait_cycles":768,"thread_start_us":93,"threads_started":1}
I20260618 22:29:36.522894 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.517963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:33889 (local address 127.16.243.130:36959)
0618 22:29:36.518076 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.518081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.518170 (+ 89us) server_negotiation.cc:407] Connection header received
0618 22:29:36.518359 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.518363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.518410 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.518458 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.518928 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.519689 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.521660 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.522286 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522373 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.522414 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.522717 (+ 303us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.522788 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":39}
I20260618 22:29:36.522984 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.517949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:34021 (local address 127.16.243.131:35905)
0618 22:29:36.518110 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.518115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.518274 (+ 159us) server_negotiation.cc:407] Connection header received
0618 22:29:36.518417 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.518419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.518459 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.518503 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.518897 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.519694 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.521405 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.521995 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522094 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.522144 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.522812 (+ 668us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.522880 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76}
I20260618 22:29:36.523687 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.517775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.16.243.129:33889)
0618 22:29:36.518076 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.518098 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.518219 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.518626 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.518629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.518646 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.518777 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.518784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.519820 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.519823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.521545 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.521552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.522467 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.522584 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.522603 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.523500 (+ 897us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.523562 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":394,"spinlock_wait_cycles":22656,"thread_start_us":80,"threads_started":1}
I20260618 22:29:36.523871 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266" is_pre_election: true
I20260618 22:29:36.524112 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.519869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:34397 (local address 127.16.243.129:33007)
0618 22:29:36.519966 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.519970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.520219 (+ 249us) server_negotiation.cc:407] Connection header received
0618 22:29:36.520413 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.520416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.520452 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.520510 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.520748 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.521361 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522843 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.523436 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.523513 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.523555 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.523953 (+ 398us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.524021 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 22:29:36.524240 31524 raft_consensus.cc:2454] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 0.
I20260618 22:29:36.524438 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.519643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33007 (local address 127.16.243.130:34397)
0618 22:29:36.520137 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.520151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.520286 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.520538 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.520541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.520559 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.520656 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.520660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.521464 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.521467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.522716 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.522725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.523719 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.523722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.523820 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.523839 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.524228 (+ 389us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.524305 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260618 22:29:36.524973 31390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075d573206ce4462a14a28ac3e8a4f95" is_pre_election: true
I20260618 22:29:36.525036 31189 leader_election.cc:302] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d029dcc7b338446dae794046815546ac, dc07fa4a091a4ab59aaea4eac6509266; no voters:
I20260618 22:29:36.525161 31390 raft_consensus.cc:2454] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 0.
I20260618 22:29:36.525332 31594 raft_consensus.cc:2781] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:36.525408 31594 raft_consensus.cc:482] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:36.525449 31594 raft_consensus.cc:3037] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:36.525476 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.520006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:41643 (local address 127.16.243.131:35905)
0618 22:29:36.520325 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.520334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.521038 (+ 704us) server_negotiation.cc:407] Connection header received
0618 22:29:36.521085 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.521088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.521139 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.521217 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:36.521484 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.522287 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.523674 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.524365 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.524905 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.525068 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.525177 (+ 109us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.525275 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
I20260618 22:29:36.525548 31256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "075d573206ce4462a14a28ac3e8a4f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d029dcc7b338446dae794046815546ac" is_pre_election: true
I20260618 22:29:36.525986 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.519869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.16.243.130:41643)
0618 22:29:36.520876 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.520888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.520966 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.521238 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.521241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.521254 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:36.521346 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.521351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.522407 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.522410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.523559 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.523568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.524568 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.524572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.524725 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.524750 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:29:36.524836 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 22:29:36.525128 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":61,"threads_started":1}
I20260618 22:29:36.526486 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "075d573206ce4462a14a28ac3e8a4f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266" is_pre_election: true
I20260618 22:29:36.526702 31524 raft_consensus.cc:2454] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075d573206ce4462a14a28ac3e8a4f95 in term 0.
I20260618 22:29:36.526793 31594 raft_consensus.cc:504] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.527009 31256 raft_consensus.cc:2379] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075d573206ce4462a14a28ac3e8a4f95 in current term 1: Already voted for candidate d029dcc7b338446dae794046815546ac in this term.
I20260618 22:29:36.527150 31594 leader_election.cc:288] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 election: Requested vote from peers 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959), dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:36.527544 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266"
I20260618 22:29:36.527642 31524 raft_consensus.cc:3037] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:36.528084 31323 leader_election.cc:302] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266; no voters:
I20260618 22:29:36.528409 31593 raft_consensus.cc:2781] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:36.528477 31593 raft_consensus.cc:482] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:36.528517 31593 raft_consensus.cc:3037] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:36.528946 31524 raft_consensus.cc:2454] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 1.
I20260618 22:29:36.529245 31390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075d573206ce4462a14a28ac3e8a4f95"
W20260618 22:29:36.529474 31303 tablet.cc:2369] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:36.529700 31593 raft_consensus.cc:504] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.530097 31593 leader_election.cc:288] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [CANDIDATE]: Term 1 election: Requested vote from peers d029dcc7b338446dae794046815546ac (127.16.243.129:33007), dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:36.530171 31189 leader_election.cc:302] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d029dcc7b338446dae794046815546ac, dc07fa4a091a4ab59aaea4eac6509266; no voters:
I20260618 22:29:36.530421 31594 raft_consensus.cc:2781] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:36.530647 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "075d573206ce4462a14a28ac3e8a4f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266"
I20260618 22:29:36.530659 31594 raft_consensus.cc:686] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 LEADER]: Becoming Leader. State: Replica: d029dcc7b338446dae794046815546ac, State: Running, Role: LEADER
I20260618 22:29:36.530658 31256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7ea3bfde854c06a9dea42c7ba4231a" candidate_uuid: "075d573206ce4462a14a28ac3e8a4f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d029dcc7b338446dae794046815546ac"
I20260618 22:29:36.530812 31524 raft_consensus.cc:2379] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 075d573206ce4462a14a28ac3e8a4f95 in current term 1: Already voted for candidate d029dcc7b338446dae794046815546ac in this term.
I20260618 22:29:36.531059 31594 consensus_queue.cc:237] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:36.531553 31390 raft_consensus.cc:2379] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d029dcc7b338446dae794046815546ac in current term 1: Already voted for candidate 075d573206ce4462a14a28ac3e8a4f95 in this term.
I20260618 22:29:36.533013 31324 leader_election.cc:302] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075d573206ce4462a14a28ac3e8a4f95; no voters: d029dcc7b338446dae794046815546ac, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:36.533452 31593 raft_consensus.cc:2726] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260618 22:29:36.534287 31109 catalog_manager.cc:5495] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac reported cstate change: term changed from 0 to 1, leader changed from <none> to d029dcc7b338446dae794046815546ac (127.16.243.129). New cstate: current_term: 1 leader_uuid: "d029dcc7b338446dae794046815546ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } health_report { overall_health: UNKNOWN } } }
I20260618 22:29:36.566529 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.562997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.16.243.129:33007)
0618 22:29:36.563129 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.563133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.563147 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:29:36.563275 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.563278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.563335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.563440 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:29:36.563687 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.564192 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.565053 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.565278 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.565368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:36.565681 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:29:36.566137 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:29:36.566249 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:36.566327 (+ 78us) server_negotiation.cc:299] Negotiation successful
0618 22:29:36.566385 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:29:36.572499 31524 raft_consensus.cc:1261] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Refusing update from remote peer d029dcc7b338446dae794046815546ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:36.572499 31390 raft_consensus.cc:1261] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Refusing update from remote peer d029dcc7b338446dae794046815546ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:36.573114 31601 consensus_queue.cc:1035] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:36.573311 31594 consensus_queue.cc:1035] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:36.580670 31606 mvcc.cc:205] Tried to move back new op lower bound from 7298341996833300480 to 7298341996673802240. Current Snapshot: MvccSnapshot[applied={T|T < 7298341996833300480}]
I20260618 22:29:36.580685 31604 mvcc.cc:205] Tried to move back new op lower bound from 7298341996833300480 to 7298341996673802240. Current Snapshot: MvccSnapshot[applied={T|T < 7298341996833300480}]
I20260618 22:29:36.589813 31109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:36.589970 31109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:36.592355 31109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Sending DeleteTablet for 3 replicas of tablet ad7ea3bfde854c06a9dea42c7ba4231a
I20260618 22:29:36.593101 31370 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:36 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:40432
I20260618 22:29:36.593147 31236 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:36 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:41968
I20260618 22:29:36.593442 31504 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:36 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:60234
I20260618 22:29:36.593691 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31173
I20260618 22:29:36.594337 31616 tablet_replica.cc:330] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: stopping tablet replica
I20260618 22:29:36.594625 31616 raft_consensus.cc:2229] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:36.594913 31616 raft_consensus.cc:2258] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:36.595479 31615 tablet_replica.cc:330] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: stopping tablet replica
I20260618 22:29:36.595865 31615 raft_consensus.cc:2229] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:36.596161 31615 raft_consensus.cc:2258] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:36.596495 31616 ts_tablet_manager.cc:1905] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:36.597466 31615 ts_tablet_manager.cc:1905] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:36.598969 31616 ts_tablet_manager.cc:1918] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:36.599045 31616 log.cc:1196] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/ad7ea3bfde854c06a9dea42c7ba4231a
I20260618 22:29:36.599295 31616 ts_tablet_manager.cc:1939] T ad7ea3bfde854c06a9dea42c7ba4231a P dc07fa4a091a4ab59aaea4eac6509266: Deleting consensus metadata
I20260618 22:29:36.600401 31095 catalog_manager.cc:4841] TS dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): tablet ad7ea3bfde854c06a9dea42c7ba4231a (table test-table [id=5b32f4c0e68b41bb900868fafcb55a47]) successfully deleted
I20260618 22:29:36.600837 31615 ts_tablet_manager.cc:1918] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260618 22:29:36.600899 31096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:33007 (error 108)
I20260618 22:29:36.601038 31615 log.cc:1196] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ad7ea3bfde854c06a9dea42c7ba4231a
I20260618 22:29:36.601107 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31307
I20260618 22:29:36.601874 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.601590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33007 (local address 127.0.0.1:41994)
0618 22:29:36.601720 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.601788 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260618 22:29:36.602033 31096 catalog_manager.cc:4584] TS d029dcc7b338446dae794046815546ac (127.16.243.129:33007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad7ea3bfde854c06a9dea42c7ba4231a: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33007: connect: Connection refused (error 111)
I20260618 22:29:36.605865 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31441
I20260618 22:29:36.606714 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.606090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.0.0.1:40448)
0618 22:29:36.606532 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.606617 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260618 22:29:36.610823 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31079
I20260618 22:29:36.616006 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--webserver_interface=127.16.243.190
--webserver_port=35803
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:36.724998 31618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:36.725337 31618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:36.725378 31618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:36.729403 31618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:36.729475 31618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:36.729493 31618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:36.729507 31618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:36.729522 31618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:36.735029 31618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=35803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:36.736109 31618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:36.737133 31618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:36.750000 31625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:36.750000 31627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:36.750020 31624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:36.750361 31618 server_base.cc:1034] running on GCE node
I20260618 22:29:36.751016 31618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:36.752213 31618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:36.753389 31618 hybrid_clock.cc:648] HybridClock initialized: now 1781821776753370 us; error 35 us; skew 500 ppm
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:36.756206 31618 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:36.757130 31618 webserver.cc:460] Webserver started at http://127.16.243.190:35803/ using document root <none> and password file <none>
I20260618 22:29:36.757726 31618 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:36.757809 31618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:36.761725 31618 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:36.763870 31634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:36.765190 31618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:29:36.765302 31618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "84007392c5bb438384faab0c6ba8aa18"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "1f74fc5c40f624b6a8916348f6531a3e"
server_key_iv: "84d16c73ccb884681a1783c1b0124848"
server_key_version: "encryptionkey@0"
I20260618 22:29:36.765933 31618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:36.782114 31618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:36.785210 31618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:36.785384 31618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:36.792488 31618 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41667
I20260618 22:29:36.792790 31686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41667 every 8 connection(s)
I20260618 22:29:36.793634 31618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:36.798103 31687 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:29:36.799963 31687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap starting.
I20260618 22:29:36.802194 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31618
I20260618 22:29:36.802719 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:33007
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=33619
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:29:36.817443 31687 log.cc:824] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:36.826350 31687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:29:36.826742 31687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap complete.
I20260618 22:29:36.830610 31687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:36.831025 31687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Initialized, Role: FOLLOWER
I20260618 22:29:36.831488 31687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:36.831597 31687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:36.831663 31687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:36.831758 31687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:29:36.833680 31687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:36.834008 31687 leader_election.cc:302] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84007392c5bb438384faab0c6ba8aa18; no voters:
I20260618 22:29:36.834288 31687 leader_election.cc:288] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:29:36.834427 31691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:29:36.834695 31691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 LEADER]: Becoming Leader. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Running, Role: LEADER
I20260618 22:29:36.835017 31691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:36.835474 31687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:36.836364 31692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:36.836491 31692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:36.836802 31693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84007392c5bb438384faab0c6ba8aa18. Latest consensus state: current_term: 2 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:36.836879 31693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:36.837271 31700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:36.842024 31700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:36.842660 31700 catalog_manager.cc:1259] Loaded cluster ID: c88ac29d8552487c881d25c66db71655
I20260618 22:29:36.842720 31700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:36.844064 31700 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260618 22:29:36.848474 31700 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260618 22:29:36.854028 31700 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:36.854887 31700 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Loaded TSK: 0
I20260618 22:29:36.856134 31700 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:29:36.919627 31689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:36.919888 31689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:36.919932 31689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:36.924007 31689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:36.924074 31689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:36.924135 31689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:36.929430 31689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=33619
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:36.930508 31689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:36.931545 31689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:36.941190 31714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:36.941192 31715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:36.941425 31689 server_base.cc:1034] running on GCE node
W20260618 22:29:36.941596 31717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:36.942022 31689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:36.942770 31689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:36.943926 31689 hybrid_clock.cc:648] HybridClock initialized: now 1781821776943895 us; error 55 us; skew 500 ppm
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:36.946830 31689 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:36.947798 31689 webserver.cc:460] Webserver started at http://127.16.243.129:33619/ using document root <none> and password file <none>
I20260618 22:29:36.948354 31689 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:36.948426 31689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:36.952423 31689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:36.954583 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:36.955878 31689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:36.955979 31689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "d029dcc7b338446dae794046815546ac"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "3d95f8e73e3b29f76844a39160295944"
server_key_iv: "797d62df73eb8f02b089603415623f60"
server_key_version: "encryptionkey@0"
I20260618 22:29:36.956517 31689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:36.970031 31689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:36.973238 31689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:36.973421 31689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:36.974030 31689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:36.975185 31731 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:29:36.979302 31689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:29:36.979367 31689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260618 22:29:36.979422 31689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:29:36.981058 31689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:29:36.981104 31689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:29:36.981371 31731 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Bootstrap starting.
I20260618 22:29:36.990664 31689 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:33007
I20260618 22:29:36.990758 31838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:33007 every 8 connection(s)
I20260618 22:29:36.991642 31689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:36.998706 31731 log.cc:824] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:36.999715 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31689
I20260618 22:29:37.000123 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:36959
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=35347
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:29:36 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:37.007083 31731 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:29:37.007443 31731 tablet_bootstrap.cc:492] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Bootstrap complete.
I20260618 22:29:37.008697 31731 ts_tablet_manager.cc:1397] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.011s
I20260618 22:29:37.008833 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.993688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:59567 (local address 127.16.243.190:41667)
0618 22:29:36.994046 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:36.994066 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:36.994384 (+ 318us) server_negotiation.cc:407] Connection header received
0618 22:29:36.995490 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:36.995501 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:36.995804 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:36.996100 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:36.997215 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.998231 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.999424 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.000052 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.000214 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.002991 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:37.002998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:37.003013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:37.003048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:37.005120 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.005783 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.005789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.005795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.005946 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.006340 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.006344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.006345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.006828 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:37.006971 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.007636 (+ 665us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.007940 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260618 22:29:37.008865 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:36.993751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:59567)
0618 22:29:36.994225 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:36.994269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:36.995087 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:36.996180 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:36.996188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:36.996598 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:36.997052 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.997066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:36.998378 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:36.998381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:36.999312 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:36.999322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.000416 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.000423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.000626 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.001298 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:37.001319 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:37.002794 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:37.005260 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.005267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.005282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.005645 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.006042 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.006045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.006047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.006229 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.006946 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:37.006953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:37.007260 (+ 307us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.007474 (+ 214us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.007837 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260618 22:29:37.011521 31839 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:37.011761 31839 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:37.011453 31731 raft_consensus.cc:348] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:37.011965 31731 raft_consensus.cc:729] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d029dcc7b338446dae794046815546ac, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.012305 31839 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
I20260618 22:29:37.012434 31731 consensus_queue.cc:260] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } } peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } }
I20260618 22:29:37.013341 31731 ts_tablet_manager.cc:1428] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260618 22:29:37.015156 31651 ts_manager.cc:194] Registered new tserver with Master: d029dcc7b338446dae794046815546ac (127.16.243.129:33007)
I20260618 22:29:37.018651 31651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:59567
I20260618 22:29:37.021397 31839 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821776, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:29:37.028251 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.018245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.16.243.129:33007)
0618 22:29:37.019027 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.019038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.019064 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:29:37.019189 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.019194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.019412 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.019551 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:37.020200 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.020909 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.021822 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.022320 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.022425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.024824 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:37.024833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:37.024844 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:37.024866 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:37.026367 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.026941 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.026946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.026948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.027031 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.027296 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.027298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.027299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.027533 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:37.027657 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.027981 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.028122 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":114,"threads_started":1}
I20260618 22:29:37.028267 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.018127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33007 (local address 127.0.0.1:41996)
0618 22:29:37.018839 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.018885 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.019028 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.019578 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.019584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.019972 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:37.020079 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.020086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.021020 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.021023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.021704 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.021715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.022439 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.022443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.022589 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.023177 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:37.023191 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:37.024671 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:37.026481 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.026490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.026496 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.026817 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.027122 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.027125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.027127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.027219 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.027625 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:37.027631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:37.027758 (+ 127us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.027893 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.028093 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":108,"threads_started":1}
I20260618 22:29:37.029268 31773 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7ea3bfde854c06a9dea42c7ba4231a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:36 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:41996
I20260618 22:29:37.030133 31848 tablet_replica.cc:330] stopping tablet replica
I20260618 22:29:37.030458 31848 raft_consensus.cc:2229] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:37.030736 31848 raft_consensus.cc:2258] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:37.031818 31848 ts_tablet_manager.cc:1905] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:37.036572 31848 ts_tablet_manager.cc:1918] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:37.036706 31848 log.cc:1196] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/ad7ea3bfde854c06a9dea42c7ba4231a
I20260618 22:29:37.036971 31848 ts_tablet_manager.cc:1939] T ad7ea3bfde854c06a9dea42c7ba4231a P d029dcc7b338446dae794046815546ac: Deleting consensus metadata
I20260618 22:29:37.039052 31636 catalog_manager.cc:4841] TS d029dcc7b338446dae794046815546ac (127.16.243.129:33007): tablet ad7ea3bfde854c06a9dea42c7ba4231a (table test-table [id=5b32f4c0e68b41bb900868fafcb55a47]) successfully deleted
W20260618 22:29:37.118983 31844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:37.119267 31844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:37.119313 31844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:37.123358 31844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:37.123430 31844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:37.123492 31844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:37.128888 31844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=35347
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:37.129951 31844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:37.131000 31844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:37.140555 31853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:37.140492 31854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.140648 31844 server_base.cc:1034] running on GCE node
W20260618 22:29:37.140507 31856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.141258 31844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:37.141893 31844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:37.143054 31844 hybrid_clock.cc:648] HybridClock initialized: now 1781821777143036 us; error 31 us; skew 500 ppm
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:37.145645 31844 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:37.146494 31844 webserver.cc:460] Webserver started at http://127.16.243.130:35347/ using document root <none> and password file <none>
I20260618 22:29:37.147076 31844 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:37.147149 31844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:37.151052 31844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:37.153467 31863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.154783 31844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:37.154901 31844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "075d573206ce4462a14a28ac3e8a4f95"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "0e6ecc5503402735f21d67b87f64bffd"
server_key_iv: "e3e940afab602e7cd7fb3f8802b26c3e"
server_key_version: "encryptionkey@0"
I20260618 22:29:37.155469 31844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:37.178145 31844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:37.181771 31844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:37.181949 31844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:37.182520 31844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:37.183898 31870 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260618 22:29:37.187893 31870 ts_tablet_manager.cc:1852] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260618 22:29:37.188305 31870 ts_tablet_manager.cc:1905] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:37.192278 31870 ts_tablet_manager.cc:1918] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:37.192400 31870 log.cc:1196] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ad7ea3bfde854c06a9dea42c7ba4231a
I20260618 22:29:37.192708 31870 ts_tablet_manager.cc:1939] T ad7ea3bfde854c06a9dea42c7ba4231a P 075d573206ce4462a14a28ac3e8a4f95: Deleting consensus metadata
I20260618 22:29:37.193011 31844 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260618 22:29:37.193080 31844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.001s
I20260618 22:29:37.193135 31844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:37.193150 31844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.202178 31844 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36959
I20260618 22:29:37.202250 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36959 every 8 connection(s)
I20260618 22:29:37.203168 31844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:37.207444 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31844
I20260618 22:29:37.207827 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:35905
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=46427
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:37.217619 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.205017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:57869 (local address 127.16.243.190:41667)
0618 22:29:37.205140 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.205143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.206083 (+ 940us) server_negotiation.cc:407] Connection header received
0618 22:29:37.207088 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.207092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.207172 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.207246 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:37.208537 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.209248 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.210425 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.210808 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.210912 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.213747 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:37.213751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:37.213754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:37.213783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:37.215262 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.215943 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.215946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.215948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.216035 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.216424 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.216426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.216427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.216613 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:37.216684 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.217362 (+ 678us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.217489 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:29:37.218815 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.205389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:57869)
0618 22:29:37.205919 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.205965 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.206867 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.207434 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.207441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.207831 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:37.208377 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.208391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.209388 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.209392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.210300 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.210310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.210983 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.210986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.211145 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.211857 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:37.211875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:37.213509 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:37.215393 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.215401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.215420 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.215820 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.216129 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.216132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.216134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.216338 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.216706 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:37.216713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:37.217075 (+ 362us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.217314 (+ 239us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.217716 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260618 22:29:37.220116 31978 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:37.220386 31978 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:37.220877 31978 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
I20260618 22:29:37.221995 31651 ts_manager.cc:194] Registered new tserver with Master: 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
I20260618 22:29:37.222806 31651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:57869
W20260618 22:29:37.319523 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:37.319802 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:37.319846 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:37.323956 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:37.324023 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:37.324082 31981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:37.329408 31981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:35905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=46427
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:37.330433 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:37.331465 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:37.341130 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.341188 31981 server_base.cc:1034] running on GCE node
W20260618 22:29:37.341121 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:37.341126 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.341750 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:37.342474 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:37.343638 31981 hybrid_clock.cc:648] HybridClock initialized: now 1781821777343617 us; error 36 us; skew 500 ppm
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:37.346302 31981 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:37.347321 31981 webserver.cc:460] Webserver started at http://127.16.243.131:46427/ using document root <none> and password file <none>
I20260618 22:29:37.347885 31981 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:37.347960 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:37.351853 31981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:37.353807 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.355116 31981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:37.355226 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "dc07fa4a091a4ab59aaea4eac6509266"
format_stamp: "Formatted at 2026-06-18 22:29:36 on dist-test-slave-jm5g"
server_key: "4dee38837dfaa5cdccd76ca7d506a5fc"
server_key_iv: "d07de0da53bc472893ac90afd76e0834"
server_key_version: "encryptionkey@0"
I20260618 22:29:37.355746 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:37.369572 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:37.372730 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:37.372907 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:37.373512 31981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:37.374408 31981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:37.374452 31981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.374495 31981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:37.374574 31981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.382961 31981 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35905
I20260618 22:29:37.382982 32109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35905 every 8 connection(s)
I20260618 22:29:37.384006 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:37.384330 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 31981
I20260618 22:29:37.393770 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.385089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.16.243.190:41667)
0618 22:29:37.385237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.385242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.385442 (+ 200us) server_negotiation.cc:407] Connection header received
0618 22:29:37.385562 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.385565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.385608 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.385681 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:37.386862 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.387622 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.388474 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.388812 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.388908 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.390386 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:37.390389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:37.390391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:37.390414 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:37.391804 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.392244 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.392246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.392247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.392315 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.392932 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.392934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.392935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.393082 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:37.393238 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.393478 (+ 240us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.393606 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":55}
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:37.398638 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.385768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:54237 (local address 127.16.243.190:41667)
0618 22:29:37.385959 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.385963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.387130 (+ 1167us) server_negotiation.cc:407] Connection header received
0618 22:29:37.388711 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.388715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.388770 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.388857 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:37.390463 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.391239 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.392562 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.392787 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.392859 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.395540 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:37.395544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:37.395546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:37.395572 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:37.396700 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.397162 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.397164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.397166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.397244 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:37.397557 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:37.397560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:37.397561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:37.397769 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:37.397857 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.398332 (+ 475us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.398444 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260618 22:29:37.399390 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.386376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:54237)
0618 22:29:37.386968 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.387023 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.388481 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.389165 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.389175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.389638 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:37.390332 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.390354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.391347 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.391351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.392468 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.392478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.392941 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.392944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.393094 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.393689 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:37.393709 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:37.395390 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:37.396796 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.396801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.396814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.397058 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.397331 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:37.397333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:37.397334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:37.397455 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:37.397861 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:37.397866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:37.398108 (+ 242us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.398263 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.398527 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":181,"threads_started":1}
I20260618 22:29:37.400449 32110 heartbeater.cc:347] Connected to a master server at 127.16.243.190:41667
I20260618 22:29:37.400660 32110 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:37.401117 32110 heartbeater.cc:511] Master 127.16.243.190:41667 requested a full tablet report, sending...
I20260618 22:29:37.402056 31651 ts_manager.cc:194] Registered new tserver with Master: dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905)
I20260618 22:29:37.402630 31651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:54237
I20260618 22:29:37.407506 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:29:37.411762 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.408089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.16.243.190:41667)
0618 22:29:37.408229 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.408233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.408385 (+ 152us) server_negotiation.cc:407] Connection header received
0618 22:29:37.408552 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.408554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.408675 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.408764 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:29:37.408986 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.409496 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.410329 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.410604 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.410701 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.410989 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:29:37.411386 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:29:37.411492 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.411559 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.411630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:29:37.412407 31651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:29:37.415709 31651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:29:37.423207 31773 tablet_service.cc:1467] Processing CreateTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 (DEFAULT_TABLE table=test-table [id=ed30663249434e8e9f3df9448657251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:37.423897 31773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2febf3cea5a144d0bf15f50b43a54105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:37.426609 31731 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Bootstrap starting.
I20260618 22:29:37.427507 31731 tablet_bootstrap.cc:654] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:37.427663 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.422256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.0.0.1:60250)
0618 22:29:37.422365 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.422380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.422451 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.423738 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.423741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.423768 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:37.423865 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.423871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.425332 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.425335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.426185 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.426195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.427217 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.427220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.427322 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.427363 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.427435 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.427521 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":15,"spinlock_wait_cycles":768}
I20260618 22:29:37.427702 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.422652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.16.243.131:35905)
0618 22:29:37.423240 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.423252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.423274 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:29:37.423366 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.423371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.423569 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.424015 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:37.424160 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.425184 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.426318 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.427104 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.427219 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.427359 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.427480 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.427570 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":138,"threads_started":1}
I20260618 22:29:37.428879 31731 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: No bootstrap required, opened a new log
I20260618 22:29:37.428978 31731 ts_tablet_manager.cc:1397] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:37.429201 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.422579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.0.0.1:40458)
0618 22:29:37.422911 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.422924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.423008 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.423711 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.423714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.423732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:37.423862 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.423870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.424997 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.425001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.427107 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.427118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.428720 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.428723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.428821 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.428847 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.428893 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.429063 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":25,"spinlock_wait_cycles":6400,"thread_start_us":119,"threads_started":1}
I20260618 22:29:37.429442 31731 raft_consensus.cc:348] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.429544 31731 raft_consensus.cc:374] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:37.429586 31731 raft_consensus.cc:729] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d029dcc7b338446dae794046815546ac, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.429701 31731 consensus_queue.cc:260] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.429958 31731 ts_tablet_manager.cc:1428] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:29:37.430015 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.422775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.16.243.130:36959)
0618 22:29:37.423228 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.423237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.423266 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:29:37.423333 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.423337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.423569 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.423727 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:37.423970 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.424749 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.427243 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.428591 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.429490 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.429616 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.429665 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.429751 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":84,"threads_started":1}
I20260618 22:29:37.431437 32044 tablet_service.cc:1467] Processing CreateTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 (DEFAULT_TABLE table=test-table [id=ed30663249434e8e9f3df9448657251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:37.431322 31912 tablet_service.cc:1467] Processing CreateTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 (DEFAULT_TABLE table=test-table [id=ed30663249434e8e9f3df9448657251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:37.432403 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2febf3cea5a144d0bf15f50b43a54105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:37.432401 31912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2febf3cea5a144d0bf15f50b43a54105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:37.437558 31870 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Bootstrap starting.
I20260618 22:29:37.439481 31870 tablet_bootstrap.cc:654] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:37.440063 32123 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Bootstrap starting.
I20260618 22:29:37.440191 31870 log.cc:824] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:37.442132 31870 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: No bootstrap required, opened a new log
I20260618 22:29:37.442298 31870 ts_tablet_manager.cc:1397] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:29:37.442615 32123 tablet_bootstrap.cc:654] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:37.443362 32123 log.cc:824] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:37.445075 32123 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: No bootstrap required, opened a new log
I20260618 22:29:37.445250 32123 ts_tablet_manager.cc:1397] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:29:37.445269 31870 raft_consensus.cc:348] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.445514 31870 raft_consensus.cc:374] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:37.445570 31870 raft_consensus.cc:729] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075d573206ce4462a14a28ac3e8a4f95, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.445981 31870 consensus_queue.cc:260] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.446697 31978 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
I20260618 22:29:37.446954 31870 ts_tablet_manager.cc:1428] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:29:37.448882 32123 raft_consensus.cc:348] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.449102 32123 raft_consensus.cc:374] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:37.449163 32123 raft_consensus.cc:729] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc07fa4a091a4ab59aaea4eac6509266, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.449604 32123 consensus_queue.cc:260] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.450404 32110 heartbeater.cc:503] Master 127.16.243.190:41667 was elected leader, sending a full tablet report...
I20260618 22:29:37.450582 32123 ts_tablet_manager.cc:1428] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260618 22:29:37.454039 31979 tablet.cc:2369] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:37.474745 31845 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:37.474965 31845 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.476063 31845 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
I20260618 22:29:37.479856 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.476523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:38379 (local address 127.16.243.131:35905)
0618 22:29:37.476643 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.476648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.476683 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:29:37.476793 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.476796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.476838 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.476894 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:37.477192 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.477685 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.478841 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.479323 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.479431 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.479481 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.479672 (+ 191us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.479741 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:29:37.479856 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.476642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:51011 (local address 127.16.243.130:36959)
0618 22:29:37.476792 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.476797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.476810 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:29:37.476854 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.476856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.476900 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.476968 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:37.477212 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.477682 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.478906 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.479313 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.479417 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.479470 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.479668 (+ 198us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.479741 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:29:37.479866 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.476277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.16.243.129:51011)
0618 22:29:37.476418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.476431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.476531 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.477003 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.477006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.477022 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:37.477107 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.477112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.477794 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.477798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.478720 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.478730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.479428 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.479431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.479526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.479569 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.479632 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.479706 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":27}
I20260618 22:29:37.479872 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.476282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.16.243.129:38379)
0618 22:29:37.476604 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.476617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.476693 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:37.476926 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:37.476928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:37.476946 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:37.477085 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.477090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.477794 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.477798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.478723 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:37.478732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.479456 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.479460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:37.479560 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.479577 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:29:37.479642 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:29:37.479722 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":115,"thread_start_us":76,"threads_started":1}
I20260618 22:29:37.480509 32064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2febf3cea5a144d0bf15f50b43a54105" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266" is_pre_election: true
I20260618 22:29:37.480513 31932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2febf3cea5a144d0bf15f50b43a54105" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075d573206ce4462a14a28ac3e8a4f95" is_pre_election: true
I20260618 22:29:37.480779 31932 raft_consensus.cc:2454] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 0.
I20260618 22:29:37.480784 32064 raft_consensus.cc:2454] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 0.
I20260618 22:29:37.481218 31725 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d029dcc7b338446dae794046815546ac, dc07fa4a091a4ab59aaea4eac6509266; no voters:
I20260618 22:29:37.481506 31845 raft_consensus.cc:2781] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:37.481585 31845 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:37.481616 31845 raft_consensus.cc:3037] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:37.482659 31845 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.483001 31845 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 election: Requested vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
I20260618 22:29:37.483359 31932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2febf3cea5a144d0bf15f50b43a54105" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075d573206ce4462a14a28ac3e8a4f95"
I20260618 22:29:37.483353 32064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2febf3cea5a144d0bf15f50b43a54105" candidate_uuid: "d029dcc7b338446dae794046815546ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc07fa4a091a4ab59aaea4eac6509266"
I20260618 22:29:37.483479 32064 raft_consensus.cc:3037] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:37.483479 31932 raft_consensus.cc:3037] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:37.484687 31932 raft_consensus.cc:2454] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 1.
I20260618 22:29:37.484687 32064 raft_consensus.cc:2454] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d029dcc7b338446dae794046815546ac in term 1.
I20260618 22:29:37.484990 31727 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 075d573206ce4462a14a28ac3e8a4f95, d029dcc7b338446dae794046815546ac, dc07fa4a091a4ab59aaea4eac6509266; no voters:
I20260618 22:29:37.485172 31845 raft_consensus.cc:2781] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:37.485379 31845 raft_consensus.cc:686] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 LEADER]: Becoming Leader. State: Replica: d029dcc7b338446dae794046815546ac, State: Running, Role: LEADER
I20260618 22:29:37.485632 31845 consensus_queue.cc:237] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.488761 31651 catalog_manager.cc:5495] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac reported cstate change: term changed from 0 to 1, leader changed from <none> to d029dcc7b338446dae794046815546ac (127.16.243.129). New cstate: current_term: 1 leader_uuid: "d029dcc7b338446dae794046815546ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } health_report { overall_health: HEALTHY } } }
W20260618 22:29:37.492817 31840 tablet.cc:2369] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:37.509574 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.506468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.16.243.129:33007)
0618 22:29:37.506625 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:37.506629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:37.506642 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:29:37.506718 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:37.506720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:37.506775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:37.506861 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:29:37.507033 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.507549 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.508348 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:37.508575 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:37.508648 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:37.508850 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:29:37.509295 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:29:37.509378 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:37.509407 (+ 29us) server_negotiation.cc:299] Negotiation successful
0618 22:29:37.509458 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:29:37.513868 31932 raft_consensus.cc:1261] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Refusing update from remote peer d029dcc7b338446dae794046815546ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:37.513868 32064 raft_consensus.cc:1261] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Refusing update from remote peer d029dcc7b338446dae794046815546ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:37.514756 31845 consensus_queue.cc:1035] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:37.514971 32129 consensus_queue.cc:1035] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:37.524650 32132 mvcc.cc:205] Tried to move back new op lower bound from 7298342000689590272 to 7298342000583720960. Current Snapshot: MvccSnapshot[applied={T|T < 7298342000689590272}]
I20260618 22:29:37.533023 31651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:37.533193 31651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:37.535630 31651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Sending DeleteTablet for 3 replicas of tablet 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:37.536450 31912 tablet_service.cc:1514] Processing DeleteTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:37 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:40458
I20260618 22:29:37.536446 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:37 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:60250
I20260618 22:29:37.536610 31773 tablet_service.cc:1514] Processing DeleteTablet for tablet 2febf3cea5a144d0bf15f50b43a54105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:37 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:41996
I20260618 22:29:37.536793 31848 tablet_replica.cc:330] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: stopping tablet replica
I20260618 22:29:37.536903 31848 raft_consensus.cc:2229] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:29:37.537222 32141 tablet_replica.cc:330] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: stopping tablet replica
I20260618 22:29:37.537251 31848 raft_consensus.cc:2258] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:37.537546 32141 raft_consensus.cc:2229] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:37.537895 32141 raft_consensus.cc:2258] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:37.536990 32142 tablet_replica.cc:330] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: stopping tablet replica
I20260618 22:29:37.538215 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31689
I20260618 22:29:37.538360 32142 raft_consensus.cc:2229] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:37.538679 32142 raft_consensus.cc:2258] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:37.540210 32141 ts_tablet_manager.cc:1905] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:37.542076 32142 ts_tablet_manager.cc:1905] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:37.542919 32141 ts_tablet_manager.cc:1918] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:37.543009 32141 log.cc:1196] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:37.543311 32141 ts_tablet_manager.cc:1939] T 2febf3cea5a144d0bf15f50b43a54105 P dc07fa4a091a4ab59aaea4eac6509266: Deleting consensus metadata
I20260618 22:29:37.544341 32142 ts_tablet_manager.cc:1918] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:37.544409 32142 log.cc:1196] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:37.544627 32142 ts_tablet_manager.cc:1939] T 2febf3cea5a144d0bf15f50b43a54105 P 075d573206ce4462a14a28ac3e8a4f95: Deleting consensus metadata
I20260618 22:29:37.544610 31635 catalog_manager.cc:4841] TS dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): tablet 2febf3cea5a144d0bf15f50b43a54105 (table test-table [id=ed30663249434e8e9f3df9448657251e]) successfully deleted
W20260618 22:29:37.545433 31636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:33007 (error 108)
I20260618 22:29:37.545701 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31844
I20260618 22:29:37.545688 31637 catalog_manager.cc:4841] TS 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): tablet 2febf3cea5a144d0bf15f50b43a54105 (table test-table [id=ed30663249434e8e9f3df9448657251e]) successfully deleted
I20260618 22:29:37.547760 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:37.547500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33007 (local address 127.0.0.1:42002)
0618 22:29:37.547604 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.547661 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 22:29:37.547909 31636 catalog_manager.cc:4584] TS d029dcc7b338446dae794046815546ac (127.16.243.129:33007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2febf3cea5a144d0bf15f50b43a54105: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33007: connect: Connection refused (error 111)
I20260618 22:29:37.550720 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31981
I20260618 22:29:37.555285 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 31618
I20260618 22:29:37.560544 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--webserver_interface=127.16.243.190
--webserver_port=35803
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:37.670838 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:37.671134 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:37.671196 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:37.675311 32144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:37.675395 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:37.675420 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:37.675451 32144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:37.675480 32144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:37.680963 32144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=35803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:37.682052 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:37.683101 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:37.691493 32149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:37.691493 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.691586 32144 server_base.cc:1034] running on GCE node
W20260618 22:29:37.691484 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.692404 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:37.693562 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:37.694723 32144 hybrid_clock.cc:648] HybridClock initialized: now 1781821777694711 us; error 29 us; skew 500 ppm
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:37.697367 32144 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:37.698225 32144 webserver.cc:460] Webserver started at http://127.16.243.190:35803/ using document root <none> and password file <none>
I20260618 22:29:37.698800 32144 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:37.698869 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:37.702613 32144 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:29:37.704661 32159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.705919 32144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:29:37.706036 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "84007392c5bb438384faab0c6ba8aa18"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "1f74fc5c40f624b6a8916348f6531a3e"
server_key_iv: "84d16c73ccb884681a1783c1b0124848"
server_key_version: "encryptionkey@0"
I20260618 22:29:37.706630 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:37.727177 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:37.730268 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:37.730451 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:37.737399 32144 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41667
I20260618 22:29:37.737694 32211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41667 every 8 connection(s)
I20260618 22:29:37.738523 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:37.742986 32212 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:29:37.744719 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap starting.
I20260618 22:29:37.746867 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 32144
I20260618 22:29:37.747330 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:33007
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=33619
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:29:37.762347 32212 log.cc:824] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:37.773769 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:29:37.774145 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18: Bootstrap complete.
I20260618 22:29:37.778024 32212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:37.778416 32212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.778916 32212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:37.779031 32212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:37.779093 32212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:37.779183 32212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:29:37.781133 32212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:37.781459 32212 leader_election.cc:302] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84007392c5bb438384faab0c6ba8aa18; no voters:
I20260618 22:29:37.781776 32212 leader_election.cc:288] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:29:37.781874 32216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:29:37.782058 32216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [term 3 LEADER]: Becoming Leader. State: Replica: 84007392c5bb438384faab0c6ba8aa18, State: Running, Role: LEADER
I20260618 22:29:37.782436 32216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } }
I20260618 22:29:37.783094 32212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:37.783540 32216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84007392c5bb438384faab0c6ba8aa18. Latest consensus state: current_term: 3 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:37.783653 32216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:37.784081 32220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:37.784029 32217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "84007392c5bb438384faab0c6ba8aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84007392c5bb438384faab0c6ba8aa18" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41667 } } }
I20260618 22:29:37.784268 32217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84007392c5bb438384faab0c6ba8aa18 [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:37.789286 32220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:37.790038 32220 catalog_manager.cc:1259] Loaded cluster ID: c88ac29d8552487c881d25c66db71655
I20260618 22:29:37.790105 32220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260618 22:29:37.791173 32220 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc0a614d9ad google::LogMessage::Fail() at ??:0
@ 0x7fc0a615088a google::LogMessage::SendToLog() at ??:0
@ 0x7fc0a614d4de google::LogMessage::Flush() at ??:0
@ 0x7fc0a614ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc0b0a92fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc0b0a93d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc0b0a90579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc0b0ad4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc0a839f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc0a839fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc0a83a13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc0a8390512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc0a99d86db start_thread at ??:0
@ 0x7fc0a4c4b61f clone at ??:0
W20260618 22:29:37.886204 32214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:37.886600 32214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:37.886715 32214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:37.892877 32214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:37.892980 32214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:37.893071 32214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:37.901145 32214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=33619
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:37.902645 32214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:37.903986 32214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:37.914943 32253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:37.914943 32254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:37.915230 32256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:37.915728 32214 server_base.cc:1034] running on GCE node
I20260618 22:29:37.916129 32214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:37.916770 32214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:37.917938 32214 hybrid_clock.cc:648] HybridClock initialized: now 1781821777917901 us; error 56 us; skew 500 ppm
Jun 18 22:29:37 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:37.920792 32214 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:37.921682 32214 webserver.cc:460] Webserver started at http://127.16.243.129:33619/ using document root <none> and password file <none>
I20260618 22:29:37.922227 32214 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:37.922297 32214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:37.926385 32214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:29:37.928778 32263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:37.930177 32214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:29:37.930301 32214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "d029dcc7b338446dae794046815546ac"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "3d95f8e73e3b29f76844a39160295944"
server_key_iv: "797d62df73eb8f02b089603415623f60"
server_key_version: "encryptionkey@0"
I20260618 22:29:37.930932 32214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:37.945495 32214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:37.948668 32214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:37.948863 32214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:37.949472 32214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:37.950572 32270 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:29:37.954653 32214 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:29:37.954715 32214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260618 22:29:37.954775 32214 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:29:37.956395 32214 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:29:37.956454 32214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:29:37.956729 32270 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Bootstrap starting.
I20260618 22:29:37.966604 32214 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:33007
I20260618 22:29:37.966944 32377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:33007 every 8 connection(s)
I20260618 22:29:37.967900 32214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:37.974391 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 32214
I20260618 22:29:37.974906 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:36959
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=35347
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:29:37.976513 32270 log.cc:824] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:37.986073 32270 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:29:37.986500 32270 tablet_bootstrap.cc:492] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Bootstrap complete.
I20260618 22:29:37.987834 32270 ts_tablet_manager.cc:1397] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260618 22:29:37.991055 32270 raft_consensus.cc:348] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.991606 32270 raft_consensus.cc:729] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d029dcc7b338446dae794046815546ac, State: Initialized, Role: FOLLOWER
I20260618 22:29:37.992129 32270 consensus_queue.cc:260] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:37.993165 32270 ts_tablet_manager.cc:1428] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260618 22:29:38.013684 32380 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:29:37.970282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:38471)
0618 22:29:37.970866 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:37.970927 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:37.972008 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:38.012959 (+ 40951us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":174,"threads_started":1}
W20260618 22:29:38.014155 32378 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:41667 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:41667: Client connection negotiation failed: client connection to 127.16.243.190:41667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:29:38.014892 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.014666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:43031)
0618 22:29:38.014777 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.014837 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:29:38.015424 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.015261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:47571)
0618 22:29:38.015343 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.015380 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:29:38.015587 32378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:29:38.091511 32381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:38.091775 32381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:38.091821 32381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:38.095835 32381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:38.095906 32381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:38.095965 32381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:38.101418 32381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:36959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=35347
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:38.102504 32381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:38.103536 32381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:38.113024 32388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:38.113092 32391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:38.113044 32389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:38.113435 32381 server_base.cc:1034] running on GCE node
I20260618 22:29:38.113878 32381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:38.114537 32381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:38.115736 32381 hybrid_clock.cc:648] HybridClock initialized: now 1781821778115712 us; error 33 us; skew 500 ppm
Jun 18 22:29:38 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:38.118363 32381 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:38.119282 32381 webserver.cc:460] Webserver started at http://127.16.243.130:35347/ using document root <none> and password file <none>
I20260618 22:29:38.119823 32381 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:38.119891 32381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:38.123705 32381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:38.125679 32398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.126941 32381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:38.127061 32381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "075d573206ce4462a14a28ac3e8a4f95"
format_stamp: "Formatted at 2026-06-18 22:29:35 on dist-test-slave-jm5g"
server_key: "0e6ecc5503402735f21d67b87f64bffd"
server_key_iv: "e3e940afab602e7cd7fb3f8802b26c3e"
server_key_version: "encryptionkey@0"
I20260618 22:29:38.127648 32381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:38.140908 32381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:38.144537 32381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:38.144761 32381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:38.145380 32381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:38.146243 32381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:38.146291 32381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.146343 32381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:38.146371 32381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.154865 32381 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36959
I20260618 22:29:38.154904 32511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36959 every 8 connection(s)
I20260618 22:29:38.155901 32381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:38.159719 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.158050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:45827)
0618 22:29:38.158524 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.158980 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
W20260618 22:29:38.160145 32512 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:41667 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:41667: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
I20260618 22:29:38.160753 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.160532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:60477)
0618 22:29:38.160643 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.160696 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 22:29:38.161278 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.161113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:53099)
0618 22:29:38.161193 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.161234 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 22:29:38.161425 32512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:29:38.162148 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 32381
I20260618 22:29:38.162497 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:35905
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:41667
--webserver_port=46427
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:38.272187 32515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:38.272473 32515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:38.272517 32515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:38.276732 32515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:38.276815 32515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:38.276876 32515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:38.282645 32515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:35905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=46427
--tserver_master_addrs=127.16.243.190:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:38.283794 32515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:38.284884 32515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:38.294364 32521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:38.294364 32520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:38.294390 32523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:38.294780 32515 server_base.cc:1034] running on GCE node
I20260618 22:29:38.295185 32515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:38.295809 32515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:38.296962 32515 hybrid_clock.cc:648] HybridClock initialized: now 1781821778296940 us; error 34 us; skew 500 ppm
Jun 18 22:29:38 dist-test-slave-jm5g krb5kdc[31066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:38.299628 32515 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:38.300518 32515 webserver.cc:460] Webserver started at http://127.16.243.131:46427/ using document root <none> and password file <none>
I20260618 22:29:38.301072 32515 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:38.301133 32515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:38.305161 32515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:38.307387 32530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.308722 32515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:38.308840 32515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "dc07fa4a091a4ab59aaea4eac6509266"
format_stamp: "Formatted at 2026-06-18 22:29:36 on dist-test-slave-jm5g"
server_key: "4dee38837dfaa5cdccd76ca7d506a5fc"
server_key_iv: "d07de0da53bc472893ac90afd76e0834"
server_key_version: "encryptionkey@0"
I20260618 22:29:38.309520 32515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:38.318936 32515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:38.321828 32515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:38.321975 32515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:38.322495 32515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:38.323313 32515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:38.323375 32515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.323413 32515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:38.323434 32515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:38.332288 32515 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35905
I20260618 22:29:38.332319 32643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35905 every 8 connection(s)
I20260618 22:29:38.333285 32515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:38.337214 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.335450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:54197)
0618 22:29:38.335942 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.336061 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
W20260618 22:29:38.337672 32644 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:41667 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:41667: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
I20260618 22:29:38.338204 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.338007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:57073)
0618 22:29:38.338104 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.338152 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 22:29:38.338579 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 32515
I20260618 22:29:38.338819 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:38.338648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:42551)
0618 22:29:38.338727 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:38.338770 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260618 22:29:38.339138 32644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:29:39.016736 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.016256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:45819)
0618 22:29:39.016560 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:39.016627 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260618 22:29:39.162467 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.162016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:50009)
0618 22:29:39.162284 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:39.162347 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260618 22:29:39.340061 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.339655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:35277)
0618 22:29:39.339899 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:39.339953 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260618 22:29:39.675346 32651 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:39.675534 32651 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:39.676569 32651 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
Jun 18 22:29:39 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.130@KRBTEST.COM
Jun 18 22:29:39 dist-test-slave-jm5g krb5kdc[31066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.131@KRBTEST.COM
I20260618 22:29:39.690249 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.676977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:40851 (local address 127.16.243.130:36959)
0618 22:29:39.677323 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:39.677339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:39.677530 (+ 191us) server_negotiation.cc:407] Connection header received
0618 22:29:39.678439 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:39.678445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:39.678856 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:39.679131 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:39.680425 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.681394 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.682594 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.683118 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.683279 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:39.685802 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:39.685808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:39.685820 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:39.685849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:39.687470 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688023 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:39.688030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:39.688035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:39.688179 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688492 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:39.688495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:39.688497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:39.688993 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:39.689104 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:39.689798 (+ 694us) server_negotiation.cc:299] Negotiation successful
0618 22:29:39.690062 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260618 22:29:39.690263 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.677129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:35839 (local address 127.16.243.131:35905)
0618 22:29:39.677473 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:39.677495 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:39.677563 (+ 68us) server_negotiation.cc:407] Connection header received
0618 22:29:39.678441 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:39.678447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:39.678856 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:39.679132 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:39.680428 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.681384 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.682600 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.683130 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.683269 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:39.686001 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:39.686006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:39.686017 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:39.686044 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:39.687632 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688036 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:39.688042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:39.688046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:39.688156 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688518 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:39.688521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:39.688522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:39.688993 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:39.689088 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:39.689771 (+ 683us) server_negotiation.cc:299] Negotiation successful
0618 22:29:39.690062 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":74,"threads_started":1}
I20260618 22:29:39.690235 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.676860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35905 (local address 127.16.243.129:35839)
0618 22:29:39.677491 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:39.677504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:39.677661 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:39.679228 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:39.679234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:39.679778 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:39.680261 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:39.680275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.681517 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.681520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:39.682457 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:39.682466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.683237 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.683241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:39.683477 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:39.684112 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:39.684123 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:39.685822 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:39.687735 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:39.687739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:39.687741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:39.687935 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:39.688246 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688257 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:39.688258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:39.688416 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:39.689169 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:39.689171 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:39.689420 (+ 249us) client_negotiation.cc:769] Sending connection context
0618 22:29:39.689647 (+ 227us) client_negotiation.cc:240] Negotiation successful
0618 22:29:39.690005 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":369,"spinlock_wait_cycles":53760,"thread_start_us":56,"threads_started":1}
I20260618 22:29:39.690235 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:39.676863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36959 (local address 127.16.243.129:40851)
0618 22:29:39.677409 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:39.677426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:39.677633 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:39.679228 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:39.679234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:39.679784 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:39.680259 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:39.680276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.681522 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.681525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:39.682456 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:39.682466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:39.683237 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:39.683240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:39.683477 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:39.684062 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:39.684078 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:39.685526 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:39.687586 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:39.687592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:39.687607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:39.687895 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:39.688291 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:39.688294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:39.688297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:39.688407 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:39.689113 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:39.689119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:39.689420 (+ 301us) client_negotiation.cc:769] Sending connection context
0618 22:29:39.689647 (+ 227us) client_negotiation.cc:240] Negotiation successful
0618 22:29:39.690067 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":241,"spinlock_wait_cycles":22400,"thread_start_us":89,"threads_started":1}
W20260618 22:29:39.692917 32266 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
W20260618 22:29:39.693078 32264 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:39.693148 32264 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d029dcc7b338446dae794046815546ac; no voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:39.693360 32651 raft_consensus.cc:2726] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:29:40.017722 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:40.017446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:39683)
0618 22:29:40.017581 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:40.017635 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:29:40.163547 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:40.163111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:55449)
0618 22:29:40.163379 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:40.163438 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260618 22:29:40.341284 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:40.340829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:39055)
0618 22:29:40.341111 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:40.341186 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260618 22:29:41.018872 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:41.018423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:58993)
0618 22:29:41.018703 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:41.018771 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260618 22:29:41.164625 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:41.164243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:46665)
0618 22:29:41.164470 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:41.164528 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260618 22:29:41.342528 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:41.342084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:50687)
0618 22:29:41.342357 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:41.342419 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260618 22:29:41.743233 32661 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:41.743379 32661 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:41.743829 32661 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
W20260618 22:29:41.744591 32264 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
W20260618 22:29:41.744707 32266 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:41.744796 32266 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d029dcc7b338446dae794046815546ac; no voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:41.744936 32661 raft_consensus.cc:2726] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:29:42.020102 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:42.019673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:58831)
0618 22:29:42.019945 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:42.020006 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260618 22:29:42.165933 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:42.165465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:57581)
0618 22:29:42.165737 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:42.165812 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260618 22:29:42.343601 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:42.343189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:36675)
0618 22:29:42.343448 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:42.343512 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260618 22:29:43.021610 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:43.021069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:57183)
0618 22:29:43.021402 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:43.021484 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":84,"threads_started":1}
I20260618 22:29:43.167145 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:43.166748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:50423)
0618 22:29:43.166987 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:43.167046 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260618 22:29:43.344928 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:43.344432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:35899)
0618 22:29:43.344759 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:43.344827 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260618 22:29:43.812394 32668 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:43.812527 32668 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:43.812944 32668 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
W20260618 22:29:43.813697 32266 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
W20260618 22:29:43.813797 32264 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:43.813894 32264 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d029dcc7b338446dae794046815546ac; no voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:43.814035 32668 raft_consensus.cc:2726] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:29:44.022893 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:44.022422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:38993)
0618 22:29:44.022712 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:44.022793 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
I20260618 22:29:44.168387 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:44.167932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:55767)
0618 22:29:44.168216 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:44.168288 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260618 22:29:44.346184 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:44.345726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:46631)
0618 22:29:44.346013 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:44.346082 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260618 22:29:45.024183 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:45.023707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:49283)
0618 22:29:45.024009 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:45.024074 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260618 22:29:45.169607 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:45.169143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:39415)
0618 22:29:45.169447 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:45.169509 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260618 22:29:45.347482 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:45.347022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:51555)
0618 22:29:45.347322 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:45.347385 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260618 22:29:46.025478 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:46.025026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:35115)
0618 22:29:46.025318 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:46.025381 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260618 22:29:46.170903 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:46.170465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:54457)
0618 22:29:46.170731 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:46.170802 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260618 22:29:46.348641 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:46.348222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:55373)
0618 22:29:46.348485 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:46.348544 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260618 22:29:46.515323 32678 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:46.515453 32678 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:46.515866 32678 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
W20260618 22:29:46.516610 32266 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
W20260618 22:29:46.516742 32264 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:46.516799 32264 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d029dcc7b338446dae794046815546ac; no voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:46.516955 32678 raft_consensus.cc:2726] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:29:47.026638 32679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:47.026198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:38639)
0618 22:29:47.026435 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:47.026500 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260618 22:29:47.172114 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:47.171684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:45735)
0618 22:29:47.171954 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:47.172014 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260618 22:29:47.349924 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:47.349427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:36259)
0618 22:29:47.349750 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:47.349822 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260618 22:29:48.028023 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:48.027562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:59129)
0618 22:29:48.027847 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:48.027922 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260618 22:29:48.173369 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:48.172930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:56401)
0618 22:29:48.173203 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:48.173271 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260618 22:29:48.351244 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:48.350806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:40445)
0618 22:29:48.351083 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:48.351144 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260618 22:29:49.029347 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:49.028866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:60925)
0618 22:29:49.029179 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:49.029248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260618 22:29:49.174505 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:49.174122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:60661)
0618 22:29:49.174353 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:49.174406 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260618 22:29:49.352355 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:49.351949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:59709)
0618 22:29:49.352189 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:49.352262 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260618 22:29:50.030489 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:50.030060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:37213)
0618 22:29:50.030334 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:50.030392 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260618 22:29:50.175750 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:50.175333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:48669)
0618 22:29:50.175589 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:50.175650 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260618 22:29:50.353621 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:50.353224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:50307)
0618 22:29:50.353465 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:50.353526 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260618 22:29:51.031853 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:51.031265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:40815)
0618 22:29:51.031606 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:51.031730 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":124,"threads_started":1}
I20260618 22:29:51.177021 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:51.176565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:54977)
0618 22:29:51.176852 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:51.176923 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260618 22:29:51.354823 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:51.354377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:55307)
0618 22:29:51.354668 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:51.354728 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260618 22:29:52.016916 32694 raft_consensus.cc:482] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:52.017062 32694 raft_consensus.cc:504] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc07fa4a091a4ab59aaea4eac6509266" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35905 } } peers { permanent_uuid: "075d573206ce4462a14a28ac3e8a4f95" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36959 } } peers { permanent_uuid: "d029dcc7b338446dae794046815546ac" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33007 } }
I20260618 22:29:52.017480 32694 leader_election.cc:288] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905), 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959)
W20260618 22:29:52.018263 32264 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc07fa4a091a4ab59aaea4eac6509266 (127.16.243.131:35905): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
W20260618 22:29:52.018425 32266 leader_election.cc:341] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 075d573206ce4462a14a28ac3e8a4f95 (127.16.243.130:36959): Not found: Tablet not found: 2febf3cea5a144d0bf15f50b43a54105
I20260618 22:29:52.018510 32266 leader_election.cc:302] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d029dcc7b338446dae794046815546ac; no voters: 075d573206ce4462a14a28ac3e8a4f95, dc07fa4a091a4ab59aaea4eac6509266
I20260618 22:29:52.018689 32694 raft_consensus.cc:2726] T 2febf3cea5a144d0bf15f50b43a54105 P d029dcc7b338446dae794046815546ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:29:52.033025 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:52.032583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:37233)
0618 22:29:52.032863 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:52.032926 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260618 22:29:52.178323 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:52.177873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:60211)
0618 22:29:52.178159 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:52.178224 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260618 22:29:52.356112 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:52.355628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.131:59003)
0618 22:29:52.355937 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:52.356009 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260618 22:29:53.034437 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:53.033967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.129:37781)
0618 22:29:53.034257 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:53.034317 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260618 22:29:53.179806 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:53.179362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41667 (local address 127.16.243.130:45841)
0618 22:29:53.179630 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:53.179695 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260618 22:29:53.346993 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 32214
I20260618 22:29:53.351850 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 32381
I20260618 22:29:53.356240 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 32515
2026-06-18T22:29:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19130 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:29:53 dist-test-slave-jm5g krb5kdc[32702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:29:53 dist-test-slave-jm5g krb5kdc[32702](info): set up 2 sockets
Jun 18 22:29:53 dist-test-slave-jm5g krb5kdc[32702](info): commencing operation
krb5kdc: starting...
W20260618 22:29:54.310432 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:29:54 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:29:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:29:54.423293 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:54.533066 32715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:54.533396 32715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:54.533466 32715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:54.537827 32715 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:29:54.537897 32715 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:29:54.537920 32715 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:29:54.537940 32715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:29:54.537962 32715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:54.537998 32715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:54.538028 32715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:29:54.538056 32715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:29:54.543624 32715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:54.544667 32715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:54.545655 32715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:29:54.554097 32715 server_base.cc:1034] running on GCE node
W20260618 22:29:54.554116 32723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:54.554126 32720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:54.554208 32721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:54.555042 32715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:54.556146 32715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:54.557296 32715 hybrid_clock.cc:648] HybridClock initialized: now 1781821794557285 us; error 37 us; skew 500 ppm
Jun 18 22:29:54 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821794, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:54.560001 32715 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:29:54.560845 32715 webserver.cc:460] Webserver started at http://127.16.243.190:44473/ using document root <none> and password file <none>
I20260618 22:29:54.561349 32715 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:54.561403 32715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:54.561569 32715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:54.563603 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "310a2201b74b4a64adf5b2d502a2d50d"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "10ffd8a5a5e8513f5805402e92320311"
server_key_iv: "a772b80feb1b2dddaa5e9d2164d63a1c"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.564105 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "310a2201b74b4a64adf5b2d502a2d50d"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "10ffd8a5a5e8513f5805402e92320311"
server_key_iv: "a772b80feb1b2dddaa5e9d2164d63a1c"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.567643 32715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:29:54.570123 32730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:54.571480 32715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:54.571616 32715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "310a2201b74b4a64adf5b2d502a2d50d"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "10ffd8a5a5e8513f5805402e92320311"
server_key_iv: "a772b80feb1b2dddaa5e9d2164d63a1c"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.571734 32715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:54.584309 32715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:54.587427 32715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:54.587590 32715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:54.593984 32715 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:45977
I20260618 22:29:54.594237 314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:45977 every 8 connection(s)
I20260618 22:29:54.594981 32715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:29:54.597671 315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:54.599462 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 32715
I20260618 22:29:54.599574 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:29:54.599820 17358 external_mini_cluster.cc:1428] Setting key 3ad5f28f8fc27b15722f6a04b818293b
I20260618 22:29:54.604341 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap starting.
I20260618 22:29:54.606896 315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:54.607745 315 log.cc:824] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:29:54 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:54.609890 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: No bootstrap required, opened a new log
I20260618 22:29:54.613884 315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:29:54.614104 315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:54.614138 315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Initialized, Role: FOLLOWER
I20260618 22:29:54.614207 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:54.601209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.16.243.190:45977)
0618 22:29:54.601610 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:54.601633 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:54.601674 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:29:54.602409 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:54.602433 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:54.602892 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:54.603285 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:54.604022 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.605431 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.606276 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.606973 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.607134 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:54.609694 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:54.609699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:54.609712 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:54.609739 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:54.611409 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:54.611882 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:54.611887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:54.611891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:54.611999 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:54.612277 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:54.612279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:54.612280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:54.612663 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:54.612814 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:54.613106 (+ 292us) server_negotiation.cc:299] Negotiation successful
0618 22:29:54.613376 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260618 22:29:54.614614 315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:29:54.614754 315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:29:54.614818 315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:29:54.614914 315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:54.616005 315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:29:54.616355 315 leader_election.cc:302] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310a2201b74b4a64adf5b2d502a2d50d; no voters:
I20260618 22:29:54.616674 315 leader_election.cc:288] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:29:54.616750 320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:54.616912 320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 LEADER]: Becoming Leader. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Running, Role: LEADER
I20260618 22:29:54.617203 320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:29:54.617908 315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:29:54.618917 322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310a2201b74b4a64adf5b2d502a2d50d. Latest consensus state: current_term: 1 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:29:54.618932 321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:29:54.619040 322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:54.619050 321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:29:54.619423 329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:29:54.622371 329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:29:54.627907 329 catalog_manager.cc:1347] Generated new cluster ID: 4b9d052c84e440bd9cd3137860bfcfd4
I20260618 22:29:54.627987 329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:29:54.655067 329 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:29:54.656046 329 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:29:54.663353 329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Generated new TSK 0
I20260618 22:29:54.664040 329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:29:54.699908 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:54.824332 341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:54.824622 341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:54.824676 341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:54.828789 341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:54.828859 341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:54.828920 341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:29:54.834290 341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:54.835417 341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:54.836474 341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:54.845999 347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:54.846001 341 server_base.cc:1034] running on GCE node
W20260618 22:29:54.845986 346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:54.845981 349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:54.846638 341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:54.847271 341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:54.848444 341 hybrid_clock.cc:648] HybridClock initialized: now 1781821794848421 us; error 36 us; skew 500 ppm
Jun 18 22:29:54 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821794, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:54.851126 341 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:29:54.851995 341 webserver.cc:460] Webserver started at http://127.16.243.129:43311/ using document root <none> and password file <none>
I20260618 22:29:54.852525 341 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:54.852592 341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:54.852788 341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:54.854861 341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "9dde377df96649c9a048600127da0e15"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "4550d8eb9b75b0fa7dd504435c07c6f3"
server_key_iv: "44400196a355fd3cd36f73179b2f3216"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.855394 341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "9dde377df96649c9a048600127da0e15"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "4550d8eb9b75b0fa7dd504435c07c6f3"
server_key_iv: "44400196a355fd3cd36f73179b2f3216"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.858944 341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:29:54.861255 356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:54.862527 341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:54.862679 341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "9dde377df96649c9a048600127da0e15"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "4550d8eb9b75b0fa7dd504435c07c6f3"
server_key_iv: "44400196a355fd3cd36f73179b2f3216"
server_key_version: "encryptionkey@0"
I20260618 22:29:54.862782 341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:54.893253 341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:54.895963 341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:54.896138 341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:54.896703 341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:54.897471 341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:54.897523 341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:54.897575 341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:54.897603 341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:54.906039 341 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:34681
I20260618 22:29:54.906067 469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:34681 every 8 connection(s)
I20260618 22:29:54.906968 341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:29:54.907186 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 341
I20260618 22:29:54.907298 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:29:54.907616 17358 external_mini_cluster.cc:1428] Setting key 6f7af2c1b15f9ad057ff2e69762decd9
Jun 18 22:29:54 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821794, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:29:54.920969 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:54.908806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:49029 (local address 127.16.243.190:45977)
0618 22:29:54.908941 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:54.908946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:54.909932 (+ 986us) server_negotiation.cc:407] Connection header received
0618 22:29:54.911488 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:54.911492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:54.911547 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:54.911631 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:54.912888 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.913414 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.914542 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.914820 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.914911 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:54.917209 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:54.917211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:54.917214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:54.917238 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:54.918784 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:54.919353 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:54.919356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:54.919358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:54.919438 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:54.919834 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:54.919838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:54.919840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:54.920040 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:54.920153 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:54.920671 (+ 518us) server_negotiation.cc:299] Negotiation successful
0618 22:29:54.920804 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:29:54.921705 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:54.909273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:49029)
0618 22:29:54.909754 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:54.909814 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:54.911277 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:54.911844 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:54.911854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:54.912265 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:54.912746 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:54.912760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.913535 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.913538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:54.914428 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:54.914437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:54.914962 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:54.914966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:54.915123 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:54.915667 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:54.915683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:54.917041 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:54.918889 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:54.918894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:54.918912 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:54.919238 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:54.919516 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:54.919519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:54.919521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:54.919727 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:54.920150 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:54.920156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:54.920402 (+ 246us) client_negotiation.cc:769] Sending connection context
0618 22:29:54.920562 (+ 160us) client_negotiation.cc:240] Negotiation successful
0618 22:29:54.920835 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
I20260618 22:29:54.922946 470 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:29:54.923177 470 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:54.923676 470 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:29:54.925164 32747 ts_manager.cc:194] Registered new tserver with Master: 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681)
I20260618 22:29:54.926261 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:49029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:29:54.933506 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:55.042200 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:55.042457 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:55.042501 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:55.047020 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:55.047094 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:55.047156 475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:29:55.052533 475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:55.053530 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:55.054468 475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:55.063953 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:55.063963 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:55.064009 475 server_base.cc:1034] running on GCE node
W20260618 22:29:55.063961 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:55.064604 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:55.065189 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:55.066359 475 hybrid_clock.cc:648] HybridClock initialized: now 1781821795066341 us; error 30 us; skew 500 ppm
Jun 18 22:29:55 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:55.069270 475 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:29:55.070067 475 webserver.cc:460] Webserver started at http://127.16.243.130:40063/ using document root <none> and password file <none>
I20260618 22:29:55.070603 475 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:55.070667 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:55.070868 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:55.072855 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "84f15f68cc15419bbd467ab4393ccef9"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "9f4213d72a96985dfb9c22ab4cd6a8a2"
server_key_iv: "999dc68b723f8beac3a4fb505e35f342"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.073393 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "84f15f68cc15419bbd467ab4393ccef9"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "9f4213d72a96985dfb9c22ab4cd6a8a2"
server_key_iv: "999dc68b723f8beac3a4fb505e35f342"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.077098 475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:29:55.079413 490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.080616 475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:29:55.080732 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "84f15f68cc15419bbd467ab4393ccef9"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "9f4213d72a96985dfb9c22ab4cd6a8a2"
server_key_iv: "999dc68b723f8beac3a4fb505e35f342"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.080832 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:55.103003 475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:55.106168 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:55.106349 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:55.106956 475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:55.107774 475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:55.107818 475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.107872 475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:55.107899 475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.115996 475 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45345
I20260618 22:29:55.116021 603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45345 every 8 connection(s)
I20260618 22:29:55.116874 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:29:55.119823 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 475
I20260618 22:29:55.119932 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:29:55.120247 17358 external_mini_cluster.cc:1428] Setting key b56839fd00bcb277d1b6088166fc8288
Jun 18 22:29:55 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:29:55.131503 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.118649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39477 (local address 127.16.243.190:45977)
0618 22:29:55.118772 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.118775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.119717 (+ 942us) server_negotiation.cc:407] Connection header received
0618 22:29:55.121264 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.121268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.121322 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.121394 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:55.122957 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.123479 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.124946 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.125274 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.125373 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.127858 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:55.127863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:55.127866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:55.127896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:55.129583 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.130081 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.130085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.130086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.130169 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.130477 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.130479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.130480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.130698 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:55.130773 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.131273 (+ 500us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.131378 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:29:55.132529 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.119115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:39477)
0618 22:29:55.119546 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.119602 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.121053 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.121582 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.121591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.122091 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:55.122819 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.122838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.123604 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.123608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.124828 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.124838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.125425 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.125428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.125575 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.126214 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:55.126230 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:55.127672 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:55.129702 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:55.129708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:55.129722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:55.129983 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:55.130234 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:55.130236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:55.130237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:55.130398 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:55.130780 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:55.130785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:55.131033 (+ 248us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.131216 (+ 183us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.131492 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260618 22:29:55.133715 604 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:29:55.133984 604 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:55.134601 604 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:29:55.135701 32747 ts_manager.cc:194] Registered new tserver with Master: 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345)
I20260618 22:29:55.136471 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:39477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:29:55.146889 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:29:55.253933 609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:29:55.254218 609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:29:55.254262 609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:29:55.258347 609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:29:55.258415 609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:29:55.258474 609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:29:55.263932 609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:29:55.264947 609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:29:55.265973 609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:29:55.275281 617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:55.275296 615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:29:55.275296 614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:29:55.275651 609 server_base.cc:1034] running on GCE node
I20260618 22:29:55.276033 609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:29:55.276691 609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:29:55.277870 609 hybrid_clock.cc:648] HybridClock initialized: now 1781821795277849 us; error 31 us; skew 500 ppm
Jun 18 22:29:55 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:29:55.280563 609 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:29:55.281395 609 webserver.cc:460] Webserver started at http://127.16.243.131:43661/ using document root <none> and password file <none>
I20260618 22:29:55.281934 609 fs_manager.cc:359] Metadata directory not provided
I20260618 22:29:55.281999 609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:29:55.282194 609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:29:55.284336 609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "5cf19dbd482144c5aa782932efd93f28"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "db14518ce55511ea917d72ebd0a9b7f1"
server_key_iv: "c6d88138a6df57d6bc3898dfa632df63"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.284889 609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "5cf19dbd482144c5aa782932efd93f28"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "db14518ce55511ea917d72ebd0a9b7f1"
server_key_iv: "c6d88138a6df57d6bc3898dfa632df63"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.288533 609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260618 22:29:55.290830 624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.292079 609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:29:55.292197 609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "5cf19dbd482144c5aa782932efd93f28"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "db14518ce55511ea917d72ebd0a9b7f1"
server_key_iv: "c6d88138a6df57d6bc3898dfa632df63"
server_key_version: "encryptionkey@0"
I20260618 22:29:55.292297 609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:29:55.307873 609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:29:55.310878 609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:29:55.311050 609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:29:55.311636 609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:29:55.312440 609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:29:55.312527 609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.312597 609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:29:55.312614 609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:29:55.320778 609 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:40935
I20260618 22:29:55.320825 737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:40935 every 8 connection(s)
I20260618 22:29:55.321674 609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:29:55.322721 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 609
I20260618 22:29:55.322844 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:29:55.323136 17358 external_mini_cluster.cc:1428] Setting key f13e7ba6cf7f3bc0bb5758c1fa839ddb
Jun 18 22:29:55 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821795, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:29:55.334705 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.323632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:33291 (local address 127.16.243.190:45977)
0618 22:29:55.323771 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.323776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.324746 (+ 970us) server_negotiation.cc:407] Connection header received
0618 22:29:55.326133 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.326136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.326182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.326244 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:55.327434 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.327984 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.329048 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.329256 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.329327 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.331670 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:55.331673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:55.331676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:55.331700 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:55.332907 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.333353 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.333356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.333357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.333422 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.333702 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.333709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.333711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.333908 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:55.333986 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.334406 (+ 420us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.334513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:29:55.335525 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.324114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:33291)
0618 22:29:55.324588 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.324646 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.325942 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.326383 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.326390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.326806 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:29:55.327299 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.327312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.328080 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.328082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.328949 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.328957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.329363 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.329365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.329498 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.330006 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:29:55.330021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:29:55.331495 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:29:55.333010 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:55.333015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:55.333029 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:55.333279 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:55.333481 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:29:55.333483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:29:55.333484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:29:55.333608 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:29:55.333983 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:29:55.333988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:29:55.334222 (+ 234us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.334374 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.334678 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260618 22:29:55.336656 738 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:29:55.336886 738 heartbeater.cc:464] Registering TS with master...
I20260618 22:29:55.337366 738 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:29:55.338272 32747 ts_manager.cc:194] Registered new tserver with Master: 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
I20260618 22:29:55.338863 32747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:33291
I20260618 22:29:55.348896 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:29:55.357894 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.351020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.16.243.190:45977)
0618 22:29:55.351156 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.351160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.351334 (+ 174us) server_negotiation.cc:407] Connection header received
0618 22:29:55.351436 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.351438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.351481 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.351544 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:29:55.352185 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.352722 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.353573 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.353797 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.353884 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.355217 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:29:55.355220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:29:55.355222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:29:55.355246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:29:55.356442 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.356837 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.356840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.356842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.356921 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:29:55.357157 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:29:55.357160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:29:55.357161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:29:55.357381 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:29:55.357492 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.357651 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.357745 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:29:55.360749 32747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:29:55.363758 32747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:29:55.378237 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34681 (local address 127.0.0.1:58062)
0618 22:29:55.372391 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.372438 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.372559 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.373761 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.373767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.373791 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:55.373908 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.373913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.375175 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.375178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.376559 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.376570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.377696 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.377700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.377810 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.377852 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.377961 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.378079 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260618 22:29:55.378271 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.16.243.129:34681)
0618 22:29:55.373246 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.373260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.373281 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:29:55.373361 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.373367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.373580 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.374037 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:55.374198 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.375062 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.376677 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.377579 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.377711 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.377845 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.378001 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.378116 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":65,"threads_started":1}
I20260618 22:29:55.378643 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.16.243.131:40935)
0618 22:29:55.373288 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.373300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.373321 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:29:55.373391 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.373396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.373595 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.373753 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:55.374190 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.375440 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.376806 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.377605 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.377711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.377845 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.378374 (+ 529us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.378475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":150,"threads_started":1}
I20260618 22:29:55.378275 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:40935 (local address 127.0.0.1:45112)
0618 22:29:55.372933 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.372946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.373031 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.373930 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.373933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.373951 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:55.374051 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.374057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.375580 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.375584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.376702 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.376709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.377906 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.377909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.378011 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.378033 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.378099 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.378161 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":12,"thread_start_us":105,"threads_started":1}
I20260618 22:29:55.379112 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.16.243.130:45345)
0618 22:29:55.373564 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.373607 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.373628 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:29:55.373699 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.373703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.373932 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.374075 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:55.374265 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.375440 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.377596 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.378482 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.378632 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.378755 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.378856 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.378949 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":87,"threads_started":1}
I20260618 22:29:55.378998 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.372337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.0.0.1:45824)
0618 22:29:55.373208 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.373220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.373293 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.374062 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.374065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.374080 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:55.374176 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.374182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.376795 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.376796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.377484 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.377490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.378615 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.378619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.378714 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.378740 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.378820 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.378884 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":79,"threads_started":1}
I20260618 22:29:55.381932 404 tablet_service.cc:1467] Processing CreateTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 (DEFAULT_TABLE table=test-table [id=962c0e2874e6454aa75192d127b39ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:55.381930 672 tablet_service.cc:1467] Processing CreateTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 (DEFAULT_TABLE table=test-table [id=962c0e2874e6454aa75192d127b39ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:55.383131 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca626c49fd64422e97d3d76448dd9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:55.383131 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca626c49fd64422e97d3d76448dd9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:55.384613 538 tablet_service.cc:1467] Processing CreateTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 (DEFAULT_TABLE table=test-table [id=962c0e2874e6454aa75192d127b39ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:29:55.385546 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca626c49fd64422e97d3d76448dd9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:29:55.390980 756 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Bootstrap starting.
I20260618 22:29:55.391618 757 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Bootstrap starting.
I20260618 22:29:55.391831 755 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Bootstrap starting.
I20260618 22:29:55.393227 756 tablet_bootstrap.cc:654] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:55.394269 755 tablet_bootstrap.cc:654] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:55.394269 757 tablet_bootstrap.cc:654] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Neither blocks nor log segments found. Creating new log.
I20260618 22:29:55.394381 756 log.cc:824] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:55.395139 757 log.cc:824] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:55.395138 755 log.cc:824] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:29:55.396566 756 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: No bootstrap required, opened a new log
I20260618 22:29:55.396764 756 ts_tablet_manager.cc:1397] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:29:55.397054 755 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: No bootstrap required, opened a new log
I20260618 22:29:55.397049 757 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: No bootstrap required, opened a new log
I20260618 22:29:55.397224 757 ts_tablet_manager.cc:1397] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:29:55.397224 755 ts_tablet_manager.cc:1397] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260618 22:29:55.399928 756 raft_consensus.cc:348] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.400164 756 raft_consensus.cc:374] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:55.400240 756 raft_consensus.cc:729] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dde377df96649c9a048600127da0e15, State: Initialized, Role: FOLLOWER
I20260618 22:29:55.400676 756 consensus_queue.cc:260] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.401055 755 raft_consensus.cc:348] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.401055 757 raft_consensus.cc:348] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.401312 757 raft_consensus.cc:374] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:55.401352 470 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
I20260618 22:29:55.401371 755 raft_consensus.cc:374] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:29:55.401422 755 raft_consensus.cc:729] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf19dbd482144c5aa782932efd93f28, State: Initialized, Role: FOLLOWER
I20260618 22:29:55.401397 757 raft_consensus.cc:729] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f15f68cc15419bbd467ab4393ccef9, State: Initialized, Role: FOLLOWER
I20260618 22:29:55.401924 755 consensus_queue.cc:260] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.401924 757 consensus_queue.cc:260] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.402784 756 ts_tablet_manager.cc:1428] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:29:55.402945 757 ts_tablet_manager.cc:1428] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:29:55.402956 755 ts_tablet_manager.cc:1428] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260618 22:29:55.403319 604 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
I20260618 22:29:55.404148 738 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
W20260618 22:29:55.408064 471 tablet.cc:2369] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:29:55.411911 761 raft_consensus.cc:482] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:29:55.412087 761 raft_consensus.cc:504] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.413148 761 leader_election.cc:288] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935), 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345)
I20260618 22:29:55.416410 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.413405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37365 (local address 127.16.243.131:40935)
0618 22:29:55.413549 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.413554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.413706 (+ 152us) server_negotiation.cc:407] Connection header received
0618 22:29:55.413748 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.413750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.413786 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.413842 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:55.414082 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.414602 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.415542 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.415917 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.416015 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.416077 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.416218 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.416288 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:29:55.416416 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.413343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:40935 (local address 127.16.243.129:37365)
0618 22:29:55.413529 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.413543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.413640 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.413864 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.413867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.413881 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:55.413981 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.413986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.414690 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.414693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.415435 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.415443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.416015 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.416019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.416117 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.416134 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.416197 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.416310 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":34}
I20260618 22:29:55.416545 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.413463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:41411 (local address 127.16.243.130:45345)
0618 22:29:55.413564 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.413568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.413788 (+ 220us) server_negotiation.cc:407] Connection header received
0618 22:29:55.413897 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.413900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.413944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.414002 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:29:55.414240 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.414738 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.415687 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.416058 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.416161 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.416208 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.416377 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.416441 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:29:55.416558 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.413365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.16.243.129:41411)
0618 22:29:55.413706 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.413717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:29:55.413805 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:29:55.414021 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:29:55.414023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:29:55.414037 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:29:55.414145 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.414150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.414838 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.414841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.415575 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:29:55.415583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.416159 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.416162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:29:55.416278 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.416297 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:29:55.416356 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 22:29:55.416421 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260618 22:29:55.417032 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca626c49fd64422e97d3d76448dd9dd3" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf19dbd482144c5aa782932efd93f28" is_pre_election: true
I20260618 22:29:55.417232 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca626c49fd64422e97d3d76448dd9dd3" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9" is_pre_election: true
I20260618 22:29:55.417296 692 raft_consensus.cc:2454] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 0.
I20260618 22:29:55.417484 558 raft_consensus.cc:2454] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 0.
I20260618 22:29:55.417817 359 leader_election.cc:302] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf19dbd482144c5aa782932efd93f28, 9dde377df96649c9a048600127da0e15; no voters:
I20260618 22:29:55.418081 761 raft_consensus.cc:2781] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:29:55.418156 761 raft_consensus.cc:482] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:29:55.418207 761 raft_consensus.cc:3037] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:55.419370 761 raft_consensus.cc:504] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.419677 761 leader_election.cc:288] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 election: Requested vote from peers 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935), 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345)
I20260618 22:29:55.420027 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca626c49fd64422e97d3d76448dd9dd3" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf19dbd482144c5aa782932efd93f28"
I20260618 22:29:55.420028 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca626c49fd64422e97d3d76448dd9dd3" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9"
I20260618 22:29:55.420141 692 raft_consensus.cc:3037] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:55.420146 558 raft_consensus.cc:3037] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:29:55.421303 692 raft_consensus.cc:2454] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 1.
I20260618 22:29:55.421331 558 raft_consensus.cc:2454] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 1.
I20260618 22:29:55.421595 359 leader_election.cc:302] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf19dbd482144c5aa782932efd93f28, 9dde377df96649c9a048600127da0e15; no voters:
I20260618 22:29:55.421763 761 raft_consensus.cc:2781] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:29:55.421983 761 raft_consensus.cc:686] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 LEADER]: Becoming Leader. State: Replica: 9dde377df96649c9a048600127da0e15, State: Running, Role: LEADER
I20260618 22:29:55.422303 761 consensus_queue.cc:237] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:29:55.425480 32747 catalog_manager.cc:5495] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dde377df96649c9a048600127da0e15 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "9dde377df96649c9a048600127da0e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } health_report { overall_health: UNKNOWN } } }
I20260618 22:29:55.441577 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.438225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.16.243.129:34681)
0618 22:29:55.438356 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:29:55.438361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:29:55.438375 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:29:55.438452 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:29:55.438454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:29:55.438507 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:29:55.438663 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:29:55.438900 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.439397 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.440206 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:29:55.440417 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:29:55.440499 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:29:55.440788 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:29:55.441203 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:29:55.441309 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:29:55.441375 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 22:29:55.441432 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:29:55.447350 558 raft_consensus.cc:1261] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Refusing update from remote peer 9dde377df96649c9a048600127da0e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:55.447350 692 raft_consensus.cc:1261] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Refusing update from remote peer 9dde377df96649c9a048600127da0e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:29:55.448107 765 consensus_queue.cc:1035] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:55.448302 761 consensus_queue.cc:1035] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:29:55.454589 768 mvcc.cc:205] Tried to move back new op lower bound from 7298342074145005568 to 7298342074052788224. Current Snapshot: MvccSnapshot[applied={T|T < 7298342074145005568}]
I20260618 22:29:55.456228 769 mvcc.cc:205] Tried to move back new op lower bound from 7298342074145005568 to 7298342074052788224. Current Snapshot: MvccSnapshot[applied={T|T < 7298342074145005568}]
I20260618 22:29:55.463958 32746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:55.464111 32746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:29:55.466450 32746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Sending DeleteTablet for 3 replicas of tablet ca626c49fd64422e97d3d76448dd9dd3
I20260618 22:29:55.467304 538 tablet_service.cc:1514] Processing DeleteTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:55 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:45824
I20260618 22:29:55.467299 672 tablet_service.cc:1514] Processing DeleteTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:55 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:45112
I20260618 22:29:55.467824 778 tablet_replica.cc:330] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: stopping tablet replica
I20260618 22:29:55.468025 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 341
I20260618 22:29:55.467994 404 tablet_service.cc:1514] Processing DeleteTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:55 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260618 22:29:55.468205 778 raft_consensus.cc:2229] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:55.468526 778 raft_consensus.cc:2258] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:55.468686 777 tablet_replica.cc:330] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: stopping tablet replica
I20260618 22:29:55.468963 777 raft_consensus.cc:2229] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:29:55.469255 777 raft_consensus.cc:2258] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:29:55.470484 778 ts_tablet_manager.cc:1905] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:55.470717 777 ts_tablet_manager.cc:1905] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:29:55.472867 777 ts_tablet_manager.cc:1918] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:55.472954 777 log.cc:1196] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/ca626c49fd64422e97d3d76448dd9dd3
I20260618 22:29:55.473223 777 ts_tablet_manager.cc:1939] T ca626c49fd64422e97d3d76448dd9dd3 P 5cf19dbd482144c5aa782932efd93f28: Deleting consensus metadata
I20260618 22:29:55.473234 778 ts_tablet_manager.cc:1918] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:29:55.473315 778 log.cc:1196] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ca626c49fd64422e97d3d76448dd9dd3
I20260618 22:29:55.473680 778 ts_tablet_manager.cc:1939] T ca626c49fd64422e97d3d76448dd9dd3 P 84f15f68cc15419bbd467ab4393ccef9: Deleting consensus metadata
I20260618 22:29:55.474431 32733 catalog_manager.cc:4841] TS 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935): tablet ca626c49fd64422e97d3d76448dd9dd3 (table test-table [id=962c0e2874e6454aa75192d127b39ed6]) successfully deleted
I20260618 22:29:55.474839 32734 catalog_manager.cc:4841] TS 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345): tablet ca626c49fd64422e97d3d76448dd9dd3 (table test-table [id=962c0e2874e6454aa75192d127b39ed6]) successfully deleted
W20260618 22:29:55.475160 32731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:34681 (error 108)
I20260618 22:29:55.475340 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 475
I20260618 22:29:55.476605 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:29:55.476305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34681 (local address 127.0.0.1:58076)
0618 22:29:55.476442 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 22:29:55.476511 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:29:55.476747 32731 catalog_manager.cc:4584] TS 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca626c49fd64422e97d3d76448dd9dd3: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34681: connect: Connection refused (error 111)
I20260618 22:29:55.480329 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 609
I20260618 22:29:55.484668 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 32715
I20260618 22:30:15.489945 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--webserver_interface=127.16.243.190
--webserver_port=44473
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:15.599820 780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:15.600121 780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:15.600179 780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:15.604347 780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:30:15.604422 780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:30:15.604444 780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:30:15.604461 780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:15.604485 780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:15.604511 780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:15.604538 780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:15.604562 780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:15.610178 780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=44473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:15.611332 780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:15.612376 780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:15.625344 789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:15.625353 786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:15.625356 787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:15.625769 780 server_base.cc:1034] running on GCE node
I20260618 22:30:15.626396 780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:15.627687 780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:15.628878 780 hybrid_clock.cc:648] HybridClock initialized: now 1781821815628856 us; error 35 us; skew 500 ppm
Jun 18 22:30:15 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:15.635432 780 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:30:15.636461 780 webserver.cc:460] Webserver started at http://127.16.243.190:44473/ using document root <none> and password file <none>
I20260618 22:30:15.637049 780 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:15.637135 780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:15.641243 780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:30:15.643441 796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:15.644714 780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:15.644838 780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "310a2201b74b4a64adf5b2d502a2d50d"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "10ffd8a5a5e8513f5805402e92320311"
server_key_iv: "a772b80feb1b2dddaa5e9d2164d63a1c"
server_key_version: "encryptionkey@0"
I20260618 22:30:15.645434 780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:15.658285 780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:15.661509 780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:15.661717 780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:15.668993 780 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:45977
I20260618 22:30:15.669253 848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:45977 every 8 connection(s)
I20260618 22:30:15.670149 780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:15.674669 849 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:30:15.676433 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 780
I20260618 22:30:15.676460 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap starting.
I20260618 22:30:15.676898 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:34681
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=43311
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:30:15.693975 849 log.cc:824] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:15.702937 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:15.703317 849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap complete.
I20260618 22:30:15.707242 849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:15.707656 849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Initialized, Role: FOLLOWER
I20260618 22:30:15.708127 849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:15.708243 849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:15.708315 849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:15.708408 849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:30:15.710305 849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:15.710675 849 leader_election.cc:302] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310a2201b74b4a64adf5b2d502a2d50d; no voters:
I20260618 22:30:15.710983 849 leader_election.cc:288] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:30:15.711120 853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:30:15.711373 853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 LEADER]: Becoming Leader. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Running, Role: LEADER
I20260618 22:30:15.711751 853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:15.712150 849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:15.713142 854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:30:15.713258 854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:15.713974 855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310a2201b74b4a64adf5b2d502a2d50d. Latest consensus state: current_term: 2 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:30:15.714423 855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:15.716840 869 catalog_manager.cc:1259] Loaded cluster ID: 4b9d052c84e440bd9cd3137860bfcfd4
I20260618 22:30:15.716892 869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: loading cluster ID for follower catalog manager: success
I20260618 22:30:15.718737 869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: acquiring CA information for follower catalog manager: success
I20260618 22:30:15.719947 869 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:30:15.720064 862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:15.722029 862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:15.722429 862 catalog_manager.cc:1259] Loaded cluster ID: 4b9d052c84e440bd9cd3137860bfcfd4
I20260618 22:30:15.722478 862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:15.723062 862 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:15.723483 862 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Loaded TSK: 0
I20260618 22:30:15.732049 862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Generated new TSK 1
I20260618 22:30:15.732306 862 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:30:15.819651 851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:15.820041 851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:15.820117 851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:15.826239 851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:15.826341 851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:15.826431 851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:15.834394 851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:34681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=43311
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:15.836001 851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:15.837472 851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:15.849181 877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:15.849200 879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:15.849270 876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:15.849782 851 server_base.cc:1034] running on GCE node
I20260618 22:30:15.850217 851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:15.850997 851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:15.852185 851 hybrid_clock.cc:648] HybridClock initialized: now 1781821815852133 us; error 97 us; skew 500 ppm
Jun 18 22:30:15 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:15.854960 851 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:30:15.855947 851 webserver.cc:460] Webserver started at http://127.16.243.129:43311/ using document root <none> and password file <none>
I20260618 22:30:15.856585 851 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:15.856675 851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:15.860860 851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:30:15.863256 886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:15.864675 851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:30:15.864794 851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "9dde377df96649c9a048600127da0e15"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "4550d8eb9b75b0fa7dd504435c07c6f3"
server_key_iv: "44400196a355fd3cd36f73179b2f3216"
server_key_version: "encryptionkey@0"
I20260618 22:30:15.865348 851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:15.893543 851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:15.896701 851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:15.896890 851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:15.897483 851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:15.898613 893 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:30:15.902781 851 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:30:15.902850 851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260618 22:30:15.902926 851 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:30:15.904752 851 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:30:15.904803 851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:30:15.905107 893 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Bootstrap starting.
I20260618 22:30:15.915107 851 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:34681
I20260618 22:30:15.915216 1000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:34681 every 8 connection(s)
I20260618 22:30:15.916121 851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:15.923449 893 log.cc:824] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:15.924546 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 851
I20260618 22:30:15.925019 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:45345
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=40063
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:30:15 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:15.932345 893 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:15.932760 893 tablet_bootstrap.cc:492] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Bootstrap complete.
I20260618 22:30:15.933910 893 ts_tablet_manager.cc:1397] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.007s
I20260618 22:30:15.934444 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:15.918151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:46625 (local address 127.16.243.190:45977)
0618 22:30:15.918599 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:15.918620 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:15.919026 (+ 406us) server_negotiation.cc:407] Connection header received
0618 22:30:15.920385 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:15.920399 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:15.920712 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:15.920981 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:15.922514 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.923547 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.924950 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.925685 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.925855 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:15.928595 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:15.928601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:15.928617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:15.928652 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:15.930760 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:15.931388 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:15.931394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:15.931399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:15.931565 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:15.931955 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:15.931958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:15.931959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:15.932414 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:15.932591 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:15.933254 (+ 663us) server_negotiation.cc:299] Negotiation successful
0618 22:30:15.933551 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260618 22:30:15.934480 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:15.918120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:46625)
0618 22:30:15.918839 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:15.918897 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:15.919973 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:15.921141 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:15.921155 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:15.921717 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:15.922359 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:15.922378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.923726 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.923729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:15.924809 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:15.924822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.925846 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.925850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:15.926052 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:15.926748 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:15.926764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:15.928360 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:15.930888 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:15.930895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:15.930911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:15.931261 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:15.931667 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:15.931670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:15.931672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:15.931853 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:15.932582 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:15.932588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:15.932909 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:30:15.933124 (+ 215us) client_negotiation.cc:240] Negotiation successful
0618 22:30:15.933477 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":125,"threads_started":1}
I20260618 22:30:15.936741 893 raft_consensus.cc:348] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:30:15.937278 893 raft_consensus.cc:729] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dde377df96649c9a048600127da0e15, State: Initialized, Role: FOLLOWER
I20260618 22:30:15.937619 1001 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:30:15.937796 893 consensus_queue.cc:260] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } }
I20260618 22:30:15.937875 1001 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:15.938421 1001 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:30:15.938812 893 ts_tablet_manager.cc:1428] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260618 22:30:15.941112 813 ts_manager.cc:194] Registered new tserver with Master: 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681)
I20260618 22:30:15.944593 813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:46625
I20260618 22:30:15.947289 1001 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
Jun 18 22:30:15 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:30:15.954916 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:15.944199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.16.243.129:34681)
0618 22:30:15.944625 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:15.944638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:15.945035 (+ 397us) server_negotiation.cc:407] Connection header received
0618 22:30:15.945218 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:15.945224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:15.945447 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:15.945588 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:15.946352 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.947470 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.948462 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.948853 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.948952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:15.951514 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:15.951523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:15.951533 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:15.951558 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:15.953005 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:15.953527 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:15.953531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:15.953533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:15.953606 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:15.953880 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:15.953883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:15.953884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:15.954129 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:15.954227 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:15.954609 (+ 382us) server_negotiation.cc:299] Negotiation successful
0618 22:30:15.954785 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20260618 22:30:15.954947 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:15.944084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34681 (local address 127.0.0.1:60948)
0618 22:30:15.944876 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:15.944921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:15.945068 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:15.945601 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:15.945607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:15.946058 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:15.946218 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:15.946228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.947606 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.947610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:15.948346 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:15.948356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:15.948973 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:15.948976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:15.949087 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:15.949665 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:15.949679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:15.951365 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:15.953119 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:15.953130 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:15.953136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:15.953431 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:15.953700 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:15.953703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:15.953705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:15.953800 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:15.954229 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:15.954235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:15.954370 (+ 135us) client_negotiation.cc:769] Sending connection context
0618 22:30:15.954516 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 22:30:15.954773 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":125,"threads_started":1}
I20260618 22:30:15.956574 935 tablet_service.cc:1514] Processing DeleteTablet for tablet ca626c49fd64422e97d3d76448dd9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:29:55 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:60948
I20260618 22:30:15.957471 1010 tablet_replica.cc:330] stopping tablet replica
I20260618 22:30:15.957763 1010 raft_consensus.cc:2229] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:15.957998 1010 raft_consensus.cc:2258] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:15.958969 1010 ts_tablet_manager.cc:1905] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:15.962527 1010 ts_tablet_manager.cc:1918] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:15.962654 1010 log.cc:1196] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal/wals/ca626c49fd64422e97d3d76448dd9dd3
I20260618 22:30:15.962881 1010 ts_tablet_manager.cc:1939] T ca626c49fd64422e97d3d76448dd9dd3 P 9dde377df96649c9a048600127da0e15: Deleting consensus metadata
I20260618 22:30:15.964620 797 catalog_manager.cc:4841] TS 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681): tablet ca626c49fd64422e97d3d76448dd9dd3 (table test-table [id=962c0e2874e6454aa75192d127b39ed6]) successfully deleted
W20260618 22:30:16.043362 1006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:16.043649 1006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:16.043712 1006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:16.047880 1006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:16.047955 1006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:16.048038 1006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:16.053485 1006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:45345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=40063
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:16.054632 1006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:16.055666 1006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:16.065330 1018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:16.065330 1015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:16.065342 1016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:16.065642 1006 server_base.cc:1034] running on GCE node
I20260618 22:30:16.066120 1006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:16.066835 1006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:16.067994 1006 hybrid_clock.cc:648] HybridClock initialized: now 1781821816067956 us; error 54 us; skew 500 ppm
Jun 18 22:30:16 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:16.070736 1006 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:30:16.071615 1006 webserver.cc:460] Webserver started at http://127.16.243.130:40063/ using document root <none> and password file <none>
I20260618 22:30:16.072175 1006 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:16.072257 1006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:16.076292 1006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:30:16.078444 1025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.079811 1006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:16.079936 1006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "84f15f68cc15419bbd467ab4393ccef9"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "9f4213d72a96985dfb9c22ab4cd6a8a2"
server_key_iv: "999dc68b723f8beac3a4fb505e35f342"
server_key_version: "encryptionkey@0"
I20260618 22:30:16.080507 1006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:16.109547 1006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:16.112975 1006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:16.113173 1006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:16.113792 1006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:16.114766 1006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:16.114815 1006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.114871 1006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:16.114899 1006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.124039 1006 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45345
I20260618 22:30:16.124068 1138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45345 every 8 connection(s)
I20260618 22:30:16.125135 1006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:16.132443 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1006
I20260618 22:30:16.132880 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:40935
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=43661
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:30:16 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:16.139740 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.127115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:50027 (local address 127.16.243.190:45977)
0618 22:30:16.127305 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.127310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.128182 (+ 872us) server_negotiation.cc:407] Connection header received
0618 22:30:16.129610 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.129613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.129677 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.129754 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:16.131084 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.131628 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.133024 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.133327 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.133430 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.136047 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:16.136051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:16.136053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:16.136080 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:16.137535 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.138092 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.138096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.138097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.138178 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.138539 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.138541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.138542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.138772 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:16.138863 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.139408 (+ 545us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.139567 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260618 22:30:16.140614 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.127541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:50027)
0618 22:30:16.128017 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.128063 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.129380 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.129950 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.129957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.130363 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:16.130945 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.130959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.131784 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.131787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.132873 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.132885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.133506 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.133510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.133685 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.134283 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:16.134301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:16.135843 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:16.137667 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:16.137675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:16.137694 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:16.137982 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:16.138262 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:16.138266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:16.138268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:16.138420 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:16.138875 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:16.138883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:16.139168 (+ 285us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.139351 (+ 183us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.139709 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260618 22:30:16.141914 1139 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:30:16.142204 1139 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:16.142863 1139 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:30:16.144124 813 ts_manager.cc:194] Registered new tserver with Master: 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345)
I20260618 22:30:16.144922 813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:50027
W20260618 22:30:16.246490 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:16.246802 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:16.246847 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:16.251135 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:16.251214 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:16.251276 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:16.256695 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:40935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=43661
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:16.257829 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:16.258852 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:16.268405 1150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:16.268420 1148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:16.268482 1147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:16.268736 1142 server_base.cc:1034] running on GCE node
I20260618 22:30:16.269165 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:16.269861 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:16.271018 1142 hybrid_clock.cc:648] HybridClock initialized: now 1781821816271004 us; error 33 us; skew 500 ppm
Jun 18 22:30:16 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:16.273672 1142 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:30:16.274578 1142 webserver.cc:460] Webserver started at http://127.16.243.131:43661/ using document root <none> and password file <none>
I20260618 22:30:16.275133 1142 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:16.275203 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:16.279179 1142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:30:16.281370 1157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.282711 1142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:30:16.282828 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "5cf19dbd482144c5aa782932efd93f28"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "db14518ce55511ea917d72ebd0a9b7f1"
server_key_iv: "c6d88138a6df57d6bc3898dfa632df63"
server_key_version: "encryptionkey@0"
I20260618 22:30:16.283385 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:16.293411 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:16.296452 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:16.296646 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:16.297240 1142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:16.298070 1142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:16.298120 1142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.298171 1142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:16.298199 1142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:16.306720 1142 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:40935
I20260618 22:30:16.306739 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:40935 every 8 connection(s)
I20260618 22:30:16.307749 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:16.309103 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1142
I20260618 22:30:16.317478 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.309814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.16.243.190:45977)
0618 22:30:16.310021 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.310025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.310108 (+ 83us) server_negotiation.cc:407] Connection header received
0618 22:30:16.310225 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.310229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.310278 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.310381 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:16.311242 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.311787 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.312888 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.313147 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.313219 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.314573 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:16.314578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:16.314581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:16.314607 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:16.315948 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.316339 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.316343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.316345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.316426 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.316673 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.316676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.316678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.316852 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:16.317056 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.317187 (+ 131us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.317308 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
Jun 18 22:30:16 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:16.321440 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.309590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:52881 (local address 127.16.243.190:45977)
0618 22:30:16.309736 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.309740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.310880 (+ 1140us) server_negotiation.cc:407] Connection header received
0618 22:30:16.312353 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.312354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.312413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.312475 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:16.313923 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.314426 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.315836 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.316043 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.316136 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.318488 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:16.318493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:16.318495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:16.318519 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:16.319742 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.320236 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.320238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.320239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.320300 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.320555 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.320557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.320558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.320729 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:16.320794 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.321235 (+ 441us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.321326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:30:16.322294 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.310139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:52881)
0618 22:30:16.310721 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.310778 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.312153 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.312722 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.312732 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.313140 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:16.313778 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.313798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.314544 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.314582 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.315723 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.315734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.316179 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.316184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.316348 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.316812 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:16.316828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:16.318339 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:16.319856 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:16.319861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:16.319875 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:16.320159 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:16.320359 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:16.320361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:16.320362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:16.320495 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:16.320794 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:16.320799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:16.321040 (+ 241us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.321208 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.321467 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260618 22:30:16.323406 1271 heartbeater.cc:347] Connected to a master server at 127.16.243.190:45977
I20260618 22:30:16.323648 1271 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:16.324131 1271 heartbeater.cc:511] Master 127.16.243.190:45977 requested a full tablet report, sending...
I20260618 22:30:16.325184 813 ts_manager.cc:194] Registered new tserver with Master: 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
I20260618 22:30:16.325748 813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:52881
I20260618 22:30:16.331364 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:30:16.335084 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.331967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.16.243.190:45977)
0618 22:30:16.332082 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.332085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.332263 (+ 178us) server_negotiation.cc:407] Connection header received
0618 22:30:16.332390 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.332392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.332492 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.332599 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:16.332771 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.333261 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.334016 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.334216 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.334281 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.334597 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:16.334748 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0618 22:30:16.334976 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32898: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:30:16.335148 1003 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32898: token expired
W20260618 22:30:16.335434 743 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.243.190:45977)
I20260618 22:30:16.341720 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.335761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.16.243.190:45977)
0618 22:30:16.335866 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.335870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.335881 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:30:16.335961 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.335963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.336000 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.336068 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:16.336581 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.337074 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.337773 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.337961 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.338021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.339198 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:16.339201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:16.339203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:16.339227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:16.340529 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.340876 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.340878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.340879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.340944 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:16.341127 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:16.341128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:16.341129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:16.341282 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:16.341373 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.341518 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.341602 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:30:16.344156 813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:30:16.347407 813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:30:16.354651 935 tablet_service.cc:1467] Processing CreateTablet for tablet dbc2d21354144e0dbdd2ad559dde14c7 (DEFAULT_TABLE table=test-table [id=a42d0cb9674748379eec63da1dc8c4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:16.355297 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc2d21354144e0dbdd2ad559dde14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:16.358093 893 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Bootstrap starting.
I20260618 22:30:16.358981 893 tablet_bootstrap.cc:654] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:16.359772 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.353666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.0.0.1:41512)
0618 22:30:16.353773 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.353790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.353866 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.355133 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.355137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.355155 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:16.355264 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.355270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.356812 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.356815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.357962 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.357972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.359339 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.359343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.359440 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.359497 (+ 57us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.359550 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.359639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 22:30:16.360179 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.354071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:40935 (local address 127.0.0.1:39714)
0618 22:30:16.355024 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.355037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.355127 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.355805 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.355808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.355826 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:16.355913 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.355920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.357212 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.357216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.358276 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.358285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.359816 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.359820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.359903 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.359924 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.359989 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.360059 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":100,"threads_started":1}
I20260618 22:30:16.360503 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.354012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.16.243.130:45345)
0618 22:30:16.354434 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.354449 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.354473 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:30:16.354579 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.354585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.354841 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.355153 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:16.355361 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.356725 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.358101 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.359210 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.360049 (+ 839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.360192 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.360248 (+ 56us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.360348 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
I20260618 22:30:16.360549 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.354512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.16.243.131:40935)
0618 22:30:16.354915 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.354926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.355388 (+ 462us) server_negotiation.cc:407] Connection header received
0618 22:30:16.355468 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.355474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.355662 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.355818 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:16.356047 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.357090 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.358403 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.359384 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.359615 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.359756 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.360335 (+ 579us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.360425 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":198,"threads_started":1}
I20260618 22:30:16.360605 893 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: No bootstrap required, opened a new log
I20260618 22:30:16.360694 893 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:30:16.361140 893 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.361272 893 raft_consensus.cc:374] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:16.361311 893 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dde377df96649c9a048600127da0e15, State: Initialized, Role: FOLLOWER
I20260618 22:30:16.361441 893 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.361691 893 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:30:16.363554 1073 tablet_service.cc:1467] Processing CreateTablet for tablet dbc2d21354144e0dbdd2ad559dde14c7 (DEFAULT_TABLE table=test-table [id=a42d0cb9674748379eec63da1dc8c4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:16.363554 1205 tablet_service.cc:1467] Processing CreateTablet for tablet dbc2d21354144e0dbdd2ad559dde14c7 (DEFAULT_TABLE table=test-table [id=a42d0cb9674748379eec63da1dc8c4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:16.364737 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc2d21354144e0dbdd2ad559dde14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:16.364745 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc2d21354144e0dbdd2ad559dde14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:16.373342 1285 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Bootstrap starting.
I20260618 22:30:16.374307 1284 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Bootstrap starting.
I20260618 22:30:16.375891 1285 tablet_bootstrap.cc:654] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:16.376987 1284 tablet_bootstrap.cc:654] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:16.377313 1285 log.cc:824] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:16.377816 1284 log.cc:824] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:16.379535 1285 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: No bootstrap required, opened a new log
I20260618 22:30:16.379652 1284 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: No bootstrap required, opened a new log
I20260618 22:30:16.379755 1285 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:30:16.379805 1284 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:30:16.383796 1284 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.383831 1285 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.384132 1284 raft_consensus.cc:374] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:16.384140 1285 raft_consensus.cc:374] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:16.384193 1285 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf19dbd482144c5aa782932efd93f28, State: Initialized, Role: FOLLOWER
I20260618 22:30:16.384193 1284 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f15f68cc15419bbd467ab4393ccef9, State: Initialized, Role: FOLLOWER
I20260618 22:30:16.384706 1284 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.384707 1285 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.385437 1271 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
I20260618 22:30:16.385675 1139 heartbeater.cc:503] Master 127.16.243.190:45977 was elected leader, sending a full tablet report...
I20260618 22:30:16.385821 1284 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260618 22:30:16.385821 1285 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260618 22:30:16.417300 1002 tablet.cc:2369] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:30:16.558801 1272 tablet.cc:2369] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:30:16.626283 1140 tablet.cc:2369] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:30:16.658689 1007 raft_consensus.cc:482] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:16.658923 1007 raft_consensus.cc:504] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.659978 1007 leader_election.cc:288] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345), 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
I20260618 22:30:16.663681 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.660353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:53061 (local address 127.16.243.130:45345)
0618 22:30:16.660482 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.660486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.660620 (+ 134us) server_negotiation.cc:407] Connection header received
0618 22:30:16.660792 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.660795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.660837 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.660896 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:16.661181 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.661670 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.662770 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.663145 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.663264 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.663318 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.663492 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.663568 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:30:16.663704 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.660220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.16.243.129:53061)
0618 22:30:16.660533 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.660546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.660666 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.660922 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.660926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.660941 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:16.661078 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.661084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.661770 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.661773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.662657 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.662667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.663264 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.663268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.663350 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.663399 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.663467 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.663543 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":21,"spinlock_wait_cycles":6912,"thread_start_us":84,"threads_started":1}
I20260618 22:30:16.663817 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.660247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:40935 (local address 127.16.243.129:57571)
0618 22:30:16.660614 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.660625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:16.660733 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:16.660948 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:16.660951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:16.660989 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:16.661092 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.661097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.661763 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.661766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.662775 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:16.662784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.663395 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.663399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:16.663479 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.663501 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:30:16.663573 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:30:16.663644 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":60,"spinlock_wait_cycles":3072,"thread_start_us":67,"threads_started":1}
I20260618 22:30:16.664141 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.660359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:57571 (local address 127.16.243.131:40935)
0618 22:30:16.660483 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.660488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.660705 (+ 217us) server_negotiation.cc:407] Connection header received
0618 22:30:16.660832 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.660835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.660873 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.660934 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:16.661190 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.661666 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.662884 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.663292 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.663395 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.663443 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.663584 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.663649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:30:16.664264 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9" is_pre_election: true
I20260618 22:30:16.664628 1093 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 0.
I20260618 22:30:16.664672 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf19dbd482144c5aa782932efd93f28" is_pre_election: true
I20260618 22:30:16.664960 1225 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 0.
I20260618 22:30:16.665162 890 leader_election.cc:302] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84f15f68cc15419bbd467ab4393ccef9, 9dde377df96649c9a048600127da0e15; no voters:
I20260618 22:30:16.665442 1007 raft_consensus.cc:2781] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:30:16.665527 1007 raft_consensus.cc:482] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:16.665575 1007 raft_consensus.cc:3037] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:16.666579 1007 raft_consensus.cc:504] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.666910 1007 leader_election.cc:288] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 election: Requested vote from peers 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345), 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
I20260618 22:30:16.667224 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9"
I20260618 22:30:16.667280 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf19dbd482144c5aa782932efd93f28"
I20260618 22:30:16.667328 1093 raft_consensus.cc:3037] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:16.667408 1225 raft_consensus.cc:3037] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:16.668422 1093 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 1.
I20260618 22:30:16.668432 1225 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 1.
I20260618 22:30:16.668712 889 leader_election.cc:302] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf19dbd482144c5aa782932efd93f28, 9dde377df96649c9a048600127da0e15; no voters:
I20260618 22:30:16.673216 1007 raft_consensus.cc:2781] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:16.673427 1007 raft_consensus.cc:686] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 LEADER]: Becoming Leader. State: Replica: 9dde377df96649c9a048600127da0e15, State: Running, Role: LEADER
I20260618 22:30:16.673689 1007 consensus_queue.cc:237] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:16.676870 812 catalog_manager.cc:5495] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dde377df96649c9a048600127da0e15 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "9dde377df96649c9a048600127da0e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } health_report { overall_health: HEALTHY } } }
I20260618 22:30:16.733817 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.730176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.16.243.129:34681)
0618 22:30:16.730491 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:16.730498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:16.730510 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:30:16.730607 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:16.730611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:16.730676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:16.730785 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:16.731042 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.731547 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.732378 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:16.732599 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:16.732693 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:16.732960 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:16.733404 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:30:16.733509 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:16.733584 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 22:30:16.733650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260618 22:30:16.737906 1093 raft_consensus.cc:1261] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Refusing update from remote peer 9dde377df96649c9a048600127da0e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:16.737906 1225 raft_consensus.cc:1261] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Refusing update from remote peer 9dde377df96649c9a048600127da0e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:16.738592 1007 consensus_queue.cc:1035] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:16.738826 1292 consensus_queue.cc:1035] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:16.747006 1297 mvcc.cc:205] Tried to move back new op lower bound from 7298342161351622656 to 7298342161098125312. Current Snapshot: MvccSnapshot[applied={T|T < 7298342161351622656}]
I20260618 22:30:16.754365 808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:16.754524 808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:16.756785 808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Sending DeleteTablet for 3 replicas of tablet dbc2d21354144e0dbdd2ad559dde14c7
I20260618 22:30:16.757506 1073 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc2d21354144e0dbdd2ad559dde14c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:16 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:41512
I20260618 22:30:16.757524 1205 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc2d21354144e0dbdd2ad559dde14c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:16 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:39714
I20260618 22:30:16.757717 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 851
I20260618 22:30:16.761719 1306 tablet_replica.cc:330] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: stopping tablet replica
I20260618 22:30:16.761722 1305 tablet_replica.cc:330] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: stopping tablet replica
I20260618 22:30:16.762084 1306 raft_consensus.cc:2229] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:16.762176 1305 raft_consensus.cc:2229] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:16.762430 1306 raft_consensus.cc:2258] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:16.762655 1305 raft_consensus.cc:2258] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 22:30:16.762954 797 connection.cc:535] client connection to 127.16.243.129:34681 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 22:30:16.763101 797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:30:16.763967 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1006
I20260618 22:30:16.764103 1306 ts_tablet_manager.cc:1905] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:16.766316 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.765992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34681 (local address 127.0.0.1:60976)
0618 22:30:16.766119 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.766207 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:30:16.766693 797 catalog_manager.cc:4584] TS 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbc2d21354144e0dbdd2ad559dde14c7: Network error: Client connection negotiation failed: client connection to 127.16.243.129:34681: connect: Connection refused (error 111)
I20260618 22:30:16.766738 1306 ts_tablet_manager.cc:1918] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:16.766822 1306 log.cc:1196] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/dbc2d21354144e0dbdd2ad559dde14c7
I20260618 22:30:16.767098 1306 ts_tablet_manager.cc:1939] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Deleting consensus metadata
I20260618 22:30:16.768203 799 catalog_manager.cc:4841] TS 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935): tablet dbc2d21354144e0dbdd2ad559dde14c7 (table test-table [id=a42d0cb9674748379eec63da1dc8c4c7]) successfully deleted
I20260618 22:30:16.769527 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1142
I20260618 22:30:16.771378 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:16.769717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.0.0.1:41528)
0618 22:30:16.771259 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:16.771311 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1459}
I20260618 22:30:16.774482 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 780
I20260618 22:30:36.779793 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--webserver_interface=127.16.243.190
--webserver_port=44473
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:36.890949 1308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:36.891240 1308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:36.891281 1308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:36.895526 1308 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:30:36.895597 1308 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:30:36.895612 1308 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:30:36.895625 1308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:36.895642 1308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:36.895659 1308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:36.895681 1308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:36.895702 1308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:36.901216 1308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=44473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:36.902307 1308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:36.903424 1308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:36.911850 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:36.911861 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:36.912060 1308 server_base.cc:1034] running on GCE node
W20260618 22:30:36.911866 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:36.912775 1308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:36.913901 1308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:36.915073 1308 hybrid_clock.cc:648] HybridClock initialized: now 1781821836915047 us; error 32 us; skew 500 ppm
Jun 18 22:30:36 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:36.922855 1308 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:30:36.923883 1308 webserver.cc:460] Webserver started at http://127.16.243.190:44473/ using document root <none> and password file <none>
I20260618 22:30:36.924475 1308 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:36.924549 1308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:36.928679 1308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:30:36.930900 1323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:36.932269 1308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:36.932394 1308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "310a2201b74b4a64adf5b2d502a2d50d"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "10ffd8a5a5e8513f5805402e92320311"
server_key_iv: "a772b80feb1b2dddaa5e9d2164d63a1c"
server_key_version: "encryptionkey@0"
I20260618 22:30:36.932976 1308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:36.953176 1308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:36.956468 1308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:36.956660 1308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:36.963560 1308 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:45977
I20260618 22:30:36.963819 1375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:45977 every 8 connection(s)
I20260618 22:30:36.964653 1308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:36.966238 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1308
I20260618 22:30:36.966754 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:34681
--local_ip_for_outbound_sockets=127.16.243.129
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=43311
--webserver_interface=127.16.243.129
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:30:36.969645 1376 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:30:36.971989 1376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap starting.
I20260618 22:30:36.993275 1376 log.cc:824] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:37.005642 1376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:37.006070 1376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Bootstrap complete.
I20260618 22:30:37.010147 1376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:37.010602 1376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Initialized, Role: FOLLOWER
I20260618 22:30:37.011096 1376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:37.011227 1376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:37.011301 1376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:37.011409 1376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:30:37.013425 1376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:37.013758 1376 leader_election.cc:302] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310a2201b74b4a64adf5b2d502a2d50d; no voters:
I20260618 22:30:37.014087 1376 leader_election.cc:288] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:30:37.014226 1380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:30:37.014451 1380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [term 3 LEADER]: Becoming Leader. State: Replica: 310a2201b74b4a64adf5b2d502a2d50d, State: Running, Role: LEADER
I20260618 22:30:37.014801 1380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } }
I20260618 22:30:37.015430 1376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:37.015813 1380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310a2201b74b4a64adf5b2d502a2d50d. Latest consensus state: current_term: 3 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:30:37.015937 1380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:37.016583 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "310a2201b74b4a64adf5b2d502a2d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310a2201b74b4a64adf5b2d502a2d50d" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 45977 } } }
I20260618 22:30:37.016692 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:37.017279 1384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:37.022622 1384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:37.023524 1384 catalog_manager.cc:1259] Loaded cluster ID: 4b9d052c84e440bd9cd3137860bfcfd4
I20260618 22:30:37.023595 1384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:37.026101 1384 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:37.026986 1384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Loaded TSK: 0
I20260618 22:30:37.027083 1384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 310a2201b74b4a64adf5b2d502a2d50d: Loaded TSK: 1
F20260618 22:30:37.027848 1384 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f16decce9ad google::LogMessage::Fail() at ??:0
@ 0x7f16decd188a google::LogMessage::SendToLog() at ??:0
@ 0x7f16decce4de google::LogMessage::Flush() at ??:0
@ 0x7f16deccfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f16e199218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f16e198af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f16e96337bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f16e960d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f16e960e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f16e964fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f16e966e82b std::function<>::operator()() at ??:0
@ 0x7f16e960db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f16e960ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f16e960b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f16e964f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f16e0f203d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f16e0f20c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f16e0f223f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f16e0f11512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f16e25556db start_thread at ??:0
@ 0x7f16dd7cc61f clone at ??:0
W20260618 22:30:37.102439 1378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:37.102759 1378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:37.102825 1378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:37.107251 1378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:37.107300 1378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:37.107366 1378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:37.113448 1378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:34681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=43311
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:37.114481 1378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:37.115506 1378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:37.125595 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:37.125684 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:37.125746 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:37.126324 1378 server_base.cc:1034] running on GCE node
I20260618 22:30:37.126820 1378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:37.127540 1378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:37.128736 1378 hybrid_clock.cc:648] HybridClock initialized: now 1781821837128709 us; error 41 us; skew 500 ppm
Jun 18 22:30:37 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:37.131690 1378 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:30:37.132608 1378 webserver.cc:460] Webserver started at http://127.16.243.129:43311/ using document root <none> and password file <none>
I20260618 22:30:37.133162 1378 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:37.133229 1378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:37.137357 1378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:30:37.139734 1433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:37.141150 1378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:30:37.141274 1378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "9dde377df96649c9a048600127da0e15"
format_stamp: "Formatted at 2026-06-18 22:29:54 on dist-test-slave-jm5g"
server_key: "4550d8eb9b75b0fa7dd504435c07c6f3"
server_key_iv: "44400196a355fd3cd36f73179b2f3216"
server_key_version: "encryptionkey@0"
I20260618 22:30:37.141853 1378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:37.164693 1378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:37.168076 1378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:37.168263 1378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:37.168879 1378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:37.170044 1440 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:30:37.175556 1378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:30:37.175657 1378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260618 22:30:37.175727 1378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:30:37.177467 1378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:30:37.177520 1378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:30:37.177845 1440 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Bootstrap starting.
I20260618 22:30:37.187366 1378 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:34681
I20260618 22:30:37.187592 1547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:34681 every 8 connection(s)
I20260618 22:30:37.188413 1378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:37.194224 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1378
I20260618 22:30:37.194671 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:45345
--local_ip_for_outbound_sockets=127.16.243.130
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=40063
--webserver_interface=127.16.243.130
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:30:37.199906 1440 log.cc:824] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:37.209925 1440 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:37.210390 1440 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Bootstrap complete.
I20260618 22:30:37.211813 1440 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Time spent bootstrapping tablet: real 0.034s user 0.032s sys 0.000s
I20260618 22:30:37.215329 1440 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:37.215873 1440 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dde377df96649c9a048600127da0e15, State: Initialized, Role: FOLLOWER
I20260618 22:30:37.216414 1440 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:37.217459 1440 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260618 22:30:37.270216 1550 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:30:37.190316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:51459)
0618 22:30:37.190819 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.190874 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:37.191703 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:37.269392 (+ 77689us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
W20260618 22:30:37.270757 1548 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45977 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:45977: Client connection negotiation failed: client connection to 127.16.243.190:45977: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:30:37.271468 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.271227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:47313)
0618 22:30:37.271348 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.271412 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 22:30:37.272064 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.271885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:57653)
0618 22:30:37.271977 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.272018 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:30:37.272223 1548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:30:37.309355 1551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:37.309614 1551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:37.309659 1551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:37.313714 1551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:37.313784 1551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:37.313844 1551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:37.319226 1551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:45345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=40063
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:37.320248 1551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:37.321252 1551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:37.330452 1558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:37.330490 1559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:37.330593 1551 server_base.cc:1034] running on GCE node
W20260618 22:30:37.330521 1561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:37.331015 1551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:37.331604 1551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:37.332778 1551 hybrid_clock.cc:648] HybridClock initialized: now 1781821837332750 us; error 39 us; skew 500 ppm
Jun 18 22:30:37 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:37.335464 1551 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:30:37.336267 1551 webserver.cc:460] Webserver started at http://127.16.243.130:40063/ using document root <none> and password file <none>
I20260618 22:30:37.336778 1551 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:37.336833 1551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:37.340559 1551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:30:37.342669 1568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:37.343870 1551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:37.343972 1551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "84f15f68cc15419bbd467ab4393ccef9"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "9f4213d72a96985dfb9c22ab4cd6a8a2"
server_key_iv: "999dc68b723f8beac3a4fb505e35f342"
server_key_version: "encryptionkey@0"
I20260618 22:30:37.344506 1551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:37.365653 1551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:37.368633 1551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:37.368791 1551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:37.369338 1551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:37.370379 1575 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:30:37.374591 1551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:30:37.374683 1551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260618 22:30:37.374758 1551 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:30:37.376456 1551 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:30:37.376506 1551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:30:37.376770 1575 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Bootstrap starting.
I20260618 22:30:37.386332 1551 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45345
I20260618 22:30:37.386412 1682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45345 every 8 connection(s)
I20260618 22:30:37.387332 1551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:37.391029 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.389223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:43977)
0618 22:30:37.389790 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.389953 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
I20260618 22:30:37.391232 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1551
W20260618 22:30:37.391538 1683 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45977 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:45977: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
I20260618 22:30:37.391693 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:40935
--local_ip_for_outbound_sockets=127.16.243.131
--tserver_master_addrs=127.16.243.190:45977
--webserver_port=43661
--webserver_interface=127.16.243.131
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:30:37.392484 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.392235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:41731)
0618 22:30:37.392364 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.392420 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:30:37.393119 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.392897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:35491)
0618 22:30:37.393001 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.393050 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:30:37.393272 1683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:30:37.394515 1575 log.cc:824] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:37.401180 1575 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:37.401553 1575 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Bootstrap complete.
I20260618 22:30:37.402484 1575 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20260618 22:30:37.405251 1575 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:37.405675 1575 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f15f68cc15419bbd467ab4393ccef9, State: Initialized, Role: FOLLOWER
I20260618 22:30:37.406098 1575 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:37.407143 1575 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260618 22:30:37.407339 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.407065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:48873)
0618 22:30:37.407194 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.407257 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:30:37.407968 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.407757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:47265)
0618 22:30:37.407858 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.407909 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:30:37.501914 1686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:37.502215 1686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:37.502260 1686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:37.506393 1686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:37.506474 1686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:37.506532 1686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:37.511955 1686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:40935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=43661
--tserver_master_addrs=127.16.243.190:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:37.512997 1686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:37.513965 1686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:37.523274 1694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:37.523289 1693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:37.523274 1696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:37.523521 1686 server_base.cc:1034] running on GCE node
I20260618 22:30:37.524001 1686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:37.524686 1686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:37.525833 1686 hybrid_clock.cc:648] HybridClock initialized: now 1781821837525813 us; error 33 us; skew 500 ppm
Jun 18 22:30:37 dist-test-slave-jm5g krb5kdc[32702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:37.528535 1686 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:30:37.529407 1686 webserver.cc:460] Webserver started at http://127.16.243.131:43661/ using document root <none> and password file <none>
I20260618 22:30:37.529929 1686 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:37.529990 1686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:37.533841 1686 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:37.536041 1703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:37.537333 1686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:30:37.537436 1686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "5cf19dbd482144c5aa782932efd93f28"
format_stamp: "Formatted at 2026-06-18 22:29:55 on dist-test-slave-jm5g"
server_key: "db14518ce55511ea917d72ebd0a9b7f1"
server_key_iv: "c6d88138a6df57d6bc3898dfa632df63"
server_key_version: "encryptionkey@0"
I20260618 22:30:37.538034 1686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:37.558826 1686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:37.562145 1686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:37.562312 1686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:37.562937 1686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:37.563766 1686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:37.563815 1686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:37.563869 1686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:37.563894 1686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:37.572427 1686 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:40935
I20260618 22:30:37.572497 1816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:40935 every 8 connection(s)
I20260618 22:30:37.573434 1686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:37.577242 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.575538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:59315)
0618 22:30:37.576002 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.576119 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
W20260618 22:30:37.577668 1817 heartbeater.cc:644] Failed to heartbeat to 127.16.243.190:45977 (0 consecutive failures): Network error: Failed to ping master at 127.16.243.190:45977: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
I20260618 22:30:37.578214 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.578007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:32841)
0618 22:30:37.578123 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.578166 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:30:37.578629 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1686
I20260618 22:30:37.578892 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:37.578681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:56439)
0618 22:30:37.578788 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:37.578836 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:30:37.579061 1817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:30:38.273227 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.272807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:50825)
0618 22:30:38.273056 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.273124 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260618 22:30:38.409076 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.408632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:50481)
0618 22:30:38.408892 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.408971 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260618 22:30:38.580127 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.579675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:34393)
0618 22:30:38.579953 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.580021 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260618 22:30:38.924285 1824 raft_consensus.cc:482] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:38.924472 1824 raft_consensus.cc:504] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:38.925490 1824 leader_election.cc:288] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345), 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
Jun 18 22:30:38 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.130@KRBTEST.COM
Jun 18 22:30:38 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.131@KRBTEST.COM
I20260618 22:30:38.938335 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.925923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:34721 (local address 127.16.243.131:40935)
0618 22:30:38.926321 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:38.926341 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:38.926378 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:30:38.927161 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:38.927170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:38.927517 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:38.927780 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:38.928743 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.929704 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.930913 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.931386 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.931528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:38.934433 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:38.934438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:38.934449 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:38.934476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:38.936017 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936478 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:38.936490 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:38.936493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:38.936600 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936843 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:38.936846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:38.936848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:38.937207 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:38.937321 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:38.937851 (+ 530us) server_negotiation.cc:299] Negotiation successful
0618 22:30:38.938136 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":75,"threads_started":1}
I20260618 22:30:38.938539 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.925925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:49789 (local address 127.16.243.130:45345)
0618 22:30:38.926466 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:38.926476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:38.926507 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:30:38.926727 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:38.926733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:38.927149 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:38.927468 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:38.928854 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.929767 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.930912 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.931385 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.931527 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:38.934094 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:38.934100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:38.934112 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:38.934138 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:38.935876 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936380 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:38.936386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:38.936392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:38.936505 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936808 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:38.936811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:38.936813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:38.937207 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:38.937320 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:38.938053 (+ 733us) server_negotiation.cc:299] Negotiation successful
0618 22:30:38.938344 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":78,"threads_started":1}
I20260618 22:30:38.938722 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.925757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:40935 (local address 127.16.243.129:34721)
0618 22:30:38.926124 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.926136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:38.926260 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:38.927739 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:38.927742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:38.928135 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:38.928578 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:38.928593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.929837 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.929839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:38.930806 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:38.930815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.931499 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.931503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:38.931733 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:38.932364 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:38.932375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:38.934248 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:38.936120 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:38.936125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:38.936373 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:38.936687 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:38.936692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:38.936767 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:38.937315 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:38.937321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:38.937566 (+ 245us) client_negotiation.cc:769] Sending connection context
0618 22:30:38.937729 (+ 163us) client_negotiation.cc:240] Negotiation successful
0618 22:30:38.938531 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":212,"spinlock_wait_cycles":24704,"thread_start_us":40,"threads_started":1}
I20260618 22:30:38.939100 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.925757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45345 (local address 127.16.243.129:49789)
0618 22:30:38.926095 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.926112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:38.926225 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:38.927599 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:38.927605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:38.928126 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:38.928577 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:38.928593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.929888 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.929890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:38.930804 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:38.930813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:38.931499 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:38.931503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:38.931734 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:38.932317 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:38.932332 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:38.933891 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:38.935988 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:38.935994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:38.936008 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:38.936278 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:38.936593 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:38.936596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:38.936599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:38.936724 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:38.937315 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:38.937321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:38.937568 (+ 247us) client_negotiation.cc:769] Sending connection context
0618 22:30:38.937730 (+ 162us) client_negotiation.cc:240] Negotiation successful
0618 22:30:38.938945 (+ 1215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":1085,"spinlock_wait_cycles":18048,"thread_start_us":65,"threads_started":1}
I20260618 22:30:38.940207 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9" is_pre_election: true
I20260618 22:30:38.940512 1637 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 1.
I20260618 22:30:38.941604 1437 leader_election.cc:302] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84f15f68cc15419bbd467ab4393ccef9, 9dde377df96649c9a048600127da0e15; no voters:
I20260618 22:30:38.941876 1824 raft_consensus.cc:2781] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:30:38.941988 1824 raft_consensus.cc:482] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:38.942039 1824 raft_consensus.cc:3037] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 1 FOLLOWER]: Advancing to term 2
W20260618 22:30:38.942422 1436 leader_election.cc:341] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935): Not found: Tablet not found: dbc2d21354144e0dbdd2ad559dde14c7
I20260618 22:30:38.944262 1824 raft_consensus.cc:504] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:38.944628 1824 leader_election.cc:288] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 election: Requested vote from peers 84f15f68cc15419bbd467ab4393ccef9 (127.16.243.130:45345), 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935)
W20260618 22:30:38.945289 1436 leader_election.cc:341] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935): Not found: Tablet not found: dbc2d21354144e0dbdd2ad559dde14c7
I20260618 22:30:38.945261 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc2d21354144e0dbdd2ad559dde14c7" candidate_uuid: "9dde377df96649c9a048600127da0e15" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "84f15f68cc15419bbd467ab4393ccef9"
I20260618 22:30:38.945403 1637 raft_consensus.cc:3037] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:30:38.947620 1637 raft_consensus.cc:2454] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dde377df96649c9a048600127da0e15 in term 2.
I20260618 22:30:38.947942 1437 leader_election.cc:302] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84f15f68cc15419bbd467ab4393ccef9, 9dde377df96649c9a048600127da0e15; no voters: 5cf19dbd482144c5aa782932efd93f28
I20260618 22:30:38.948107 1824 raft_consensus.cc:2781] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:30:38.948290 1824 raft_consensus.cc:686] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [term 2 LEADER]: Becoming Leader. State: Replica: 9dde377df96649c9a048600127da0e15, State: Running, Role: LEADER
I20260618 22:30:38.948635 1824 consensus_queue.cc:237] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:38.949357 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:38.948793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:48779)
0618 22:30:38.948973 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:38.949269 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260618 22:30:39.328816 1637 raft_consensus.cc:1261] T dbc2d21354144e0dbdd2ad559dde14c7 P 84f15f68cc15419bbd467ab4393ccef9 [term 2 FOLLOWER]: Refusing update from remote peer 9dde377df96649c9a048600127da0e15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260618 22:30:39.329510 1824 consensus_queue.cc:1035] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260618 22:30:39.331416 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.331018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:38527)
0618 22:30:39.331235 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.331304 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
W20260618 22:30:39.333787 1436 consensus_peers.cc:487] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15 -> Peer 5cf19dbd482144c5aa782932efd93f28 (127.16.243.131:40935): Couldn't send request to peer 5cf19dbd482144c5aa782932efd93f28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbc2d21354144e0dbdd2ad559dde14c7. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:30:39.334033 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.333620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:46579)
0618 22:30:39.333938 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.333977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260618 22:30:39.334674 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.334442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:33881)
0618 22:30:39.334542 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.334611 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 22:30:39.581459 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.580979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:45429)
0618 22:30:39.581271 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.581342 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260618 22:30:39.781445 1838 ts_tablet_manager.cc:927] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Initiating tablet copy from peer 9dde377df96649c9a048600127da0e15 (127.16.243.129:34681)
I20260618 22:30:39.782073 1838 tablet_copy_client.cc:316] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.243.129:34681
Jun 18 22:30:39 dist-test-slave-jm5g krb5kdc[32702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.129@KRBTEST.COM
I20260618 22:30:39.792110 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.782613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:34681 (local address 127.16.243.131:35125)
0618 22:30:39.782742 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.782767 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:39.782905 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:39.783592 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:39.783598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:39.783988 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:39.784109 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:39.784115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:39.785137 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:39.785140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:39.785769 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:39.785777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:39.786354 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:39.786357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:39.786463 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:39.787004 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:39.787016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:39.788354 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:39.790289 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:39.790299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:39.790305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:39.790536 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:39.790877 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:39.790880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:39.790882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:39.790962 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:39.791478 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:39.791486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:39.791621 (+ 135us) client_negotiation.cc:769] Sending connection context
0618 22:30:39.791740 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 22:30:39.791921 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 22:30:39.792160 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.782778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:35125 (local address 127.16.243.129:34681)
0618 22:30:39.783108 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:39.783114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:39.783149 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:30:39.783198 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:39.783204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:39.783441 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:39.783588 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:39.784226 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:39.785024 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:39.785882 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:39.786240 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:39.786334 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:39.788503 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:39.788511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:39.788524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:39.788549 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:39.790175 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:39.790681 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:39.790688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:39.790690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:39.790778 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:39.791047 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:39.791050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:39.791052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:39.791375 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:39.791497 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:39.791854 (+ 357us) server_negotiation.cc:299] Negotiation successful
0618 22:30:39.792004 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":64,"threads_started":1}
I20260618 22:30:39.793241 1520 tablet_copy_service.cc:140] P 9dde377df96649c9a048600127da0e15: Received BeginTabletCopySession request for tablet dbc2d21354144e0dbdd2ad559dde14c7 from peer 5cf19dbd482144c5aa782932efd93f28 ({username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:35125)
I20260618 22:30:39.793408 1520 tablet_copy_service.cc:161] P 9dde377df96649c9a048600127da0e15: Beginning new tablet copy session on tablet dbc2d21354144e0dbdd2ad559dde14c7 from peer 5cf19dbd482144c5aa782932efd93f28 at {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:35125: session id = 5cf19dbd482144c5aa782932efd93f28-dbc2d21354144e0dbdd2ad559dde14c7
I20260618 22:30:39.794828 1520 tablet_copy_source_session.cc:205] T dbc2d21354144e0dbdd2ad559dde14c7 P 9dde377df96649c9a048600127da0e15: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:30:39.798766 1838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc2d21354144e0dbdd2ad559dde14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:39.805184 1838 tablet_copy_client.cc:791] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: tablet copy: Starting download of 0 data blocks...
I20260618 22:30:39.805384 1838 tablet_copy_client.cc:655] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: tablet copy: Starting download of 1 WAL segments...
I20260618 22:30:39.807197 1838 tablet_copy_client.cc:531] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:30:39.809019 1838 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Bootstrap starting.
I20260618 22:30:39.814451 1838 log.cc:824] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:39.821480 1838 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:30:39.821789 1838 tablet_bootstrap.cc:492] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Bootstrap complete.
I20260618 22:30:39.822093 1838 ts_tablet_manager.cc:1397] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260618 22:30:39.824541 1838 raft_consensus.cc:348] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:39.824806 1838 raft_consensus.cc:729] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf19dbd482144c5aa782932efd93f28, State: Initialized, Role: FOLLOWER
I20260618 22:30:39.825191 1838 consensus_queue.cc:260] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84f15f68cc15419bbd467ab4393ccef9" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45345 } } peers { permanent_uuid: "5cf19dbd482144c5aa782932efd93f28" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 40935 } } peers { permanent_uuid: "9dde377df96649c9a048600127da0e15" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 34681 } }
I20260618 22:30:39.826018 1838 ts_tablet_manager.cc:1428] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260618 22:30:39.826503 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.826186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:52817)
0618 22:30:39.826357 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.826420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260618 22:30:39.826778 1520 tablet_copy_service.cc:342] P 9dde377df96649c9a048600127da0e15: Request end of tablet copy session 5cf19dbd482144c5aa782932efd93f28-dbc2d21354144e0dbdd2ad559dde14c7 received from {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:35125
I20260618 22:30:39.826905 1520 tablet_copy_service.cc:433] P 9dde377df96649c9a048600127da0e15: ending tablet copy session 5cf19dbd482144c5aa782932efd93f28-dbc2d21354144e0dbdd2ad559dde14c7 on tablet dbc2d21354144e0dbdd2ad559dde14c7 with peer 5cf19dbd482144c5aa782932efd93f28
I20260618 22:30:39.827173 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:39.826981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:40177)
0618 22:30:39.827074 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:39.827115 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 22:30:40.197746 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:40.197389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:41907)
0618 22:30:40.197600 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:40.197656 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260618 22:30:40.197786 1771 raft_consensus.cc:1203] T dbc2d21354144e0dbdd2ad559dde14c7 P 5cf19dbd482144c5aa782932efd93f28 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260618 22:30:40.198777 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:40.198484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:48767)
0618 22:30:40.198642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:40.198703 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 22:30:40.335743 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:40.335310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:55273)
0618 22:30:40.335591 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:40.335650 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260618 22:30:41.198817 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:41.198387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:44749)
0618 22:30:41.198669 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:41.198723 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260618 22:30:41.199745 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:41.199370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:35417)
0618 22:30:41.199613 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:41.199664 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":58,"threads_started":1}
I20260618 22:30:41.336984 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:41.336536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:54115)
0618 22:30:41.336822 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:41.336887 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260618 22:30:42.200157 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:42.199731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:44775)
0618 22:30:42.199992 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:42.200056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260618 22:30:42.200820 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:42.200628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:57111)
0618 22:30:42.200722 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:42.200766 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 22:30:42.200871 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:42.200428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:55669)
0618 22:30:42.200731 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:42.200789 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":70,"threads_started":1}
I20260618 22:30:42.338078 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:42.337657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:48551)
0618 22:30:42.337929 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:42.337989 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260618 22:30:43.202062 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:43.201705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:35973)
0618 22:30:43.201927 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:43.201978 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":54,"threads_started":1}
I20260618 22:30:43.202059 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:43.201595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:45373)
0618 22:30:43.201890 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:43.201957 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260618 22:30:43.339385 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:43.338974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:58545)
0618 22:30:43.339231 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:43.339292 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260618 22:30:44.203365 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:44.202917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:54125)
0618 22:30:44.203199 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:44.203265 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260618 22:30:44.203404 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:44.202917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:54537)
0618 22:30:44.203266 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:44.203321 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":88,"threads_started":1}
I20260618 22:30:44.340458 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:44.340044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:44249)
0618 22:30:44.340310 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:44.340366 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
I20260618 22:30:45.204538 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:45.204089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:43207)
0618 22:30:45.204373 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:45.204432 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260618 22:30:45.204538 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:45.204089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:42393)
0618 22:30:45.204373 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:45.204432 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260618 22:30:45.341615 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:45.341190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:45229)
0618 22:30:45.341459 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:45.341518 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260618 22:30:46.205750 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:46.205315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:43399)
0618 22:30:46.205578 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:46.205654 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260618 22:30:46.205760 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:46.205376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:49447)
0618 22:30:46.205616 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:46.205673 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260618 22:30:46.342844 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:46.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:40595)
0618 22:30:46.342682 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:46.342743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260618 22:30:47.206986 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:47.206582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:54123)
0618 22:30:47.206831 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:47.206890 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260618 22:30:47.207026 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:47.206581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:57183)
0618 22:30:47.206892 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:47.206941 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":69,"threads_started":1}
I20260618 22:30:47.343869 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:47.343490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:45079)
0618 22:30:47.343723 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:47.343777 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260618 22:30:48.208118 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:48.207724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:58585)
0618 22:30:48.207961 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:48.208017 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260618 22:30:48.208177 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:48.207724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:54423)
0618 22:30:48.208029 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:48.208093 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":83,"threads_started":1}
I20260618 22:30:48.345125 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:48.344650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:49885)
0618 22:30:48.344961 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:48.345026 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260618 22:30:49.209355 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:49.208929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:53477)
0618 22:30:49.209193 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:49.209258 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260618 22:30:49.209422 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:49.208929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:42693)
0618 22:30:49.209284 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:49.209338 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":85,"threads_started":1}
I20260618 22:30:49.346320 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:49.345885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:47331)
0618 22:30:49.346153 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:49.346223 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260618 22:30:50.210533 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:50.210132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:42391)
0618 22:30:50.210381 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:50.210438 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260618 22:30:50.210619 1876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:50.210200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:50561)
0618 22:30:50.210434 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:50.210486 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260618 22:30:50.347404 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:50.346991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:34189)
0618 22:30:50.347234 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:50.347290 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260618 22:30:51.211710 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:51.211258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:44945)
0618 22:30:51.211523 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:51.211591 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
I20260618 22:30:51.211756 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:51.211277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:44161)
0618 22:30:51.211619 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:51.211676 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
I20260618 22:30:51.348515 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:51.348060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:38199)
0618 22:30:51.348314 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:51.348401 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260618 22:30:52.213021 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:52.212599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.131:38975)
0618 22:30:52.212863 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:52.212925 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260618 22:30:52.213057 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:52.212606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.129:44465)
0618 22:30:52.212913 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:52.212962 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20260618 22:30:52.349743 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:52.349304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:45977 (local address 127.16.243.130:48083)
0618 22:30:52.349561 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:52.349640 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.190:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260618 22:30:52.581005 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1378
I20260618 22:30:52.585947 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1551
I20260618 22:30:52.590538 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1686
2026-06-18T22:30:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59234 ms)
[----------] 30 tests from SecurityITest (235864 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:30:52 dist-test-slave-jm5g krb5kdc[1886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:30:52 dist-test-slave-jm5g krb5kdc[1886](info): set up 2 sockets
Jun 18 22:30:52 dist-test-slave-jm5g krb5kdc[1886](info): commencing operation
krb5kdc: starting...
W20260618 22:30:53.538208 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:30:53 dist-test-slave-jm5g krb5kdc[1886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:30:53.651765 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43939
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:53.760262 1899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:53.760563 1899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:53.760607 1899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:53.764602 1899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:53.764663 1899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:53.764681 1899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:53.764698 1899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:53.764711 1899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:53.769992 1899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:53.771068 1899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:53.772044 1899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:30:53.780654 1899 server_base.cc:1034] running on GCE node
W20260618 22:30:53.780642 1904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:53.780861 1907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:53.780642 1905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:53.781469 1899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:53.782600 1899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:53.783767 1899 hybrid_clock.cc:648] HybridClock initialized: now 1781821853783753 us; error 33 us; skew 500 ppm
Jun 18 22:30:53 dist-test-slave-jm5g krb5kdc[1886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821853, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:53.786523 1899 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:30:53.787552 1899 webserver.cc:460] Webserver started at http://127.16.243.190:40735/ using document root <none> and password file <none>
I20260618 22:30:53.788131 1899 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:53.788199 1899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:53.788403 1899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:53.790482 1899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "41dc3d6f5d1a43838a7c8f20777f965e"
format_stamp: "Formatted at 2026-06-18 22:30:53 on dist-test-slave-jm5g"
server_key: "56360706cabbca1c8e09edbbad1402f8"
server_key_iv: "e80a64f9d6c6fecfd6acd41d788df155"
server_key_version: "encryptionkey@0"
I20260618 22:30:53.791072 1899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "41dc3d6f5d1a43838a7c8f20777f965e"
format_stamp: "Formatted at 2026-06-18 22:30:53 on dist-test-slave-jm5g"
server_key: "56360706cabbca1c8e09edbbad1402f8"
server_key_iv: "e80a64f9d6c6fecfd6acd41d788df155"
server_key_version: "encryptionkey@0"
I20260618 22:30:53.794811 1899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:30:53.797147 1914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:53.798370 1899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:53.798485 1899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "41dc3d6f5d1a43838a7c8f20777f965e"
format_stamp: "Formatted at 2026-06-18 22:30:53 on dist-test-slave-jm5g"
server_key: "56360706cabbca1c8e09edbbad1402f8"
server_key_iv: "e80a64f9d6c6fecfd6acd41d788df155"
server_key_version: "encryptionkey@0"
I20260618 22:30:53.798643 1899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:53.818041 1899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:53.820981 1899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:53.821180 1899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:53.828226 1899 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:43939
I20260618 22:30:53.828238 1966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:43939 every 8 connection(s)
I20260618 22:30:53.829190 1899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:53.832295 1967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:53.837528 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1899
I20260618 22:30:53.837656 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:30:53.837713 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: Bootstrap starting.
I20260618 22:30:53.837917 17358 external_mini_cluster.cc:1428] Setting key 7c1c2d2ce091e036a423c791873e28d2
I20260618 22:30:53.840353 1967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:53.841037 1967 log.cc:824] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:53.842921 1967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: No bootstrap required, opened a new log
I20260618 22:30:53.846774 1967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } }
I20260618 22:30:53.847069 1967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:53.847138 1967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41dc3d6f5d1a43838a7c8f20777f965e, State: Initialized, Role: FOLLOWER
Jun 18 22:30:53 dist-test-slave-jm5g krb5kdc[1886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:53.847589 1967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } }
I20260618 22:30:53.847712 1967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:53.847772 1967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:53.847860 1967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:53.848879 1967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } }
I20260618 22:30:53.849205 1967 leader_election.cc:302] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41dc3d6f5d1a43838a7c8f20777f965e; no voters:
I20260618 22:30:53.849493 1967 leader_election.cc:288] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:30:53.849588 1972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:53.849745 1972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [term 1 LEADER]: Becoming Leader. State: Replica: 41dc3d6f5d1a43838a7c8f20777f965e, State: Running, Role: LEADER
I20260618 22:30:53.850099 1972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } }
I20260618 22:30:53.850800 1967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:53.852345 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:53.839339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.16.243.190:43939)
0618 22:30:53.839878 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:53.839923 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:53.839968 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:30:53.840729 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:53.840750 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:53.841095 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:53.841417 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:53.842242 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:53.843340 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:53.844229 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:53.844989 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:53.845157 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:53.847714 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:53.847721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:53.847738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:53.847772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:53.849405 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:53.849903 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:53.849908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:53.849914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:53.850037 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:53.850293 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:53.850297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:53.850298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:53.850796 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:53.850985 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:53.851248 (+ 263us) server_negotiation.cc:299] Negotiation successful
0618 22:30:53.851542 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":133,"threads_started":1}
I20260618 22:30:53.853175 1974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41dc3d6f5d1a43838a7c8f20777f965e. Latest consensus state: current_term: 1 leader_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } } }
I20260618 22:30:53.853381 1974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:53.855154 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41dc3d6f5d1a43838a7c8f20777f965e" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43939 } } }
I20260618 22:30:53.855234 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e [sys.catalog]: This master's current role is: LEADER
W20260618 22:30:53.855743 1986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:30:53.855819 1986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:30:53.855880 1987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:53.856608 1987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:53.862097 1987 catalog_manager.cc:1347] Generated new cluster ID: bd496212b5fe421ba39f930da1a24231
I20260618 22:30:53.862170 1987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:53.876389 1987 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:30:53.877358 1987 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:53.887236 1987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: Generated new TSK 0
I20260618 22:30:53.888000 1987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:30:53.913565 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:54.024785 1993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:54.025070 1993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:54.025115 1993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:54.029166 1993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:54.029230 1993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:54.029292 1993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:54.035059 1993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:54.036135 1993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:54.037101 1993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:54.046259 1998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.046322 1999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.046273 2001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:54.046597 1993 server_base.cc:1034] running on GCE node
I20260618 22:30:54.046998 1993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:54.047636 1993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:54.048789 1993 hybrid_clock.cc:648] HybridClock initialized: now 1781821854048770 us; error 31 us; skew 500 ppm
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:54.051505 1993 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:30:54.052450 1993 webserver.cc:460] Webserver started at http://127.16.243.129:32901/ using document root <none> and password file <none>
I20260618 22:30:54.052999 1993 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:54.053062 1993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:54.053246 1993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:54.055342 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "d5fffd52e9cf4e6a95c82369c3857d21"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cb9b84eed51b0948097d224495b0b5e0"
server_key_iv: "e66dbc7bbb3c9d33f5fc07502e0ae582"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.055835 1993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "d5fffd52e9cf4e6a95c82369c3857d21"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cb9b84eed51b0948097d224495b0b5e0"
server_key_iv: "e66dbc7bbb3c9d33f5fc07502e0ae582"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.059449 1993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:30:54.061770 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.063012 1993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:30:54.063153 1993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "d5fffd52e9cf4e6a95c82369c3857d21"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cb9b84eed51b0948097d224495b0b5e0"
server_key_iv: "e66dbc7bbb3c9d33f5fc07502e0ae582"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.063259 1993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:54.088721 1993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:54.091946 1993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:54.092139 1993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:54.092729 1993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:54.093587 1993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:54.093636 1993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.093688 1993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:54.093714 1993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.102044 1993 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:33457
I20260618 22:30:54.102073 2121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:33457 every 8 connection(s)
I20260618 22:30:54.103026 1993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:54.110009 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 1993
I20260618 22:30:54.110143 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:30:54.110451 17358 external_mini_cluster.cc:1428] Setting key e1b1aec4ff3123622357086ebf9a9fca
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:54.116730 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.104763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:41335 (local address 127.16.243.190:43939)
0618 22:30:54.104889 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.104892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.105773 (+ 881us) server_negotiation.cc:407] Connection header received
0618 22:30:54.107154 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.107157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.107218 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.107289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:54.108479 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.109011 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.110354 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.110657 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.110745 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.113344 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:54.113348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:54.113350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:54.113375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:54.114760 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.115243 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.115246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.115247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.115324 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.115634 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.115638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.115640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.115822 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:54.115906 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.116425 (+ 519us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.116554 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:30:54.117547 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.105133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43939 (local address 127.16.243.129:41335)
0618 22:30:54.105608 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.105654 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.106933 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.107479 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.107487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.107839 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:54.108334 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.108347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.109130 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.109132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.110233 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.110245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.110815 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.110819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.110974 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.111596 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:54.111612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:54.113182 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:54.114867 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.114873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.114890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.115152 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.115406 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.115409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.115411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.115547 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.115915 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:54.115922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:54.116187 (+ 265us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.116374 (+ 187us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.116713 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260618 22:30:54.118788 2122 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43939
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:30:54.119076 2122 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:54.119612 2122 heartbeater.cc:511] Master 127.16.243.190:43939 requested a full tablet report, sending...
I20260618 22:30:54.121084 1931 ts_manager.cc:194] Registered new tserver with Master: d5fffd52e9cf4e6a95c82369c3857d21 (127.16.243.129:33457)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:30:54.122200 1931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:41335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:30:54.135293 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:54.246181 2127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:54.246457 2127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:54.246517 2127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:54.250535 2127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:54.250626 2127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:54.250701 2127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:54.256031 2127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:54.257076 2127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:54.258069 2127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:54.267253 2132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.267308 2133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.267343 2135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:54.267295 2127 server_base.cc:1034] running on GCE node
I20260618 22:30:54.267905 2127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:54.268535 2127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:54.269696 2127 hybrid_clock.cc:648] HybridClock initialized: now 1781821854269676 us; error 36 us; skew 500 ppm
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:54.272374 2127 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:30:54.273249 2127 webserver.cc:460] Webserver started at http://127.16.243.130:34211/ using document root <none> and password file <none>
I20260618 22:30:54.273772 2127 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:54.273829 2127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:54.273998 2127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:54.276000 2127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "6fc878032a824f5789b948f5dd40a67e"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cde353e9c1976bfbad0259fd387a2448"
server_key_iv: "0d61ba3d5e212db2e6575173c0e0858e"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.276506 2127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "6fc878032a824f5789b948f5dd40a67e"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cde353e9c1976bfbad0259fd387a2448"
server_key_iv: "0d61ba3d5e212db2e6575173c0e0858e"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.280126 2127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:30:54.282442 2142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.283684 2127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:54.283792 2127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "6fc878032a824f5789b948f5dd40a67e"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "cde353e9c1976bfbad0259fd387a2448"
server_key_iv: "0d61ba3d5e212db2e6575173c0e0858e"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.283885 2127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:54.314379 2127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:54.317636 2127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:54.317816 2127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:54.318414 2127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:54.319294 2127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:54.319350 2127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.319404 2127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:54.319432 2127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.327837 2127 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36983
I20260618 22:30:54.327862 2255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36983 every 8 connection(s)
I20260618 22:30:54.328691 2127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:54.331513 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2127
I20260618 22:30:54.331651 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:30:54.331965 17358 external_mini_cluster.cc:1428] Setting key e7c979c3ebbd41d1872873d712500e62
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:30:54.342923 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.330406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:53033 (local address 127.16.243.190:43939)
0618 22:30:54.330592 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.330597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.331498 (+ 901us) server_negotiation.cc:407] Connection header received
0618 22:30:54.333048 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.333051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.333105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.333186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:54.334633 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.335340 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.336438 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.336672 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.336751 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.339373 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:54.339377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:54.339380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:54.339411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:54.340913 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.341399 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.341402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.341404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.341500 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.341867 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.341870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.341872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.342095 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:54.342174 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.342695 (+ 521us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.342802 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:30:54.343816 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.330857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43939 (local address 127.16.243.130:53033)
0618 22:30:54.331318 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.331373 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.332829 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.333364 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.333373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.333835 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:54.334488 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.334506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.335458 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.335461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.336336 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.336344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.336819 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.336823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.337007 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.337566 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:54.337587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:54.339211 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:54.341021 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.341026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.341038 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.341290 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.341580 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.341583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.341586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.341707 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.342184 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:54.342189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:54.342423 (+ 234us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.342632 (+ 209us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.342940 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:30:54.344955 2256 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43939
I20260618 22:30:54.345189 2256 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:54.345716 2256 heartbeater.cc:511] Master 127.16.243.190:43939 requested a full tablet report, sending...
I20260618 22:30:54.346877 1931 ts_manager.cc:194] Registered new tserver with Master: 6fc878032a824f5789b948f5dd40a67e (127.16.243.130:36983)
I20260618 22:30:54.347445 1931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:53033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:30:54.357401 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:54.467631 2261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:54.467913 2261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:54.467972 2261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:54.471969 2261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:54.472045 2261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:54.472138 2261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:54.477519 2261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:54.478596 2261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:54.479604 2261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:54.489284 2269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.489293 2266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:54.489346 2261 server_base.cc:1034] running on GCE node
W20260618 22:30:54.489300 2267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:54.489893 2261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:54.490605 2261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:54.491793 2261 hybrid_clock.cc:648] HybridClock initialized: now 1781821854491774 us; error 30 us; skew 500 ppm
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:54.494609 2261 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:30:54.495473 2261 webserver.cc:460] Webserver started at http://127.16.243.131:42027/ using document root <none> and password file <none>
I20260618 22:30:54.496021 2261 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:54.496086 2261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:54.496269 2261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:54.498279 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "ef170f1624e44964b353db76b0180826"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "d9eb7d0b3790bc1024aed520fe847e79"
server_key_iv: "0c7c8b2c8aff5ac54e545e54720546c3"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.498831 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "ef170f1624e44964b353db76b0180826"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "d9eb7d0b3790bc1024aed520fe847e79"
server_key_iv: "0c7c8b2c8aff5ac54e545e54720546c3"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.502579 2261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:30:54.505170 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.506386 2261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:54.506497 2261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "ef170f1624e44964b353db76b0180826"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "d9eb7d0b3790bc1024aed520fe847e79"
server_key_iv: "0c7c8b2c8aff5ac54e545e54720546c3"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.506632 2261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:54.525480 2261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:54.529047 2261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:54.529255 2261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:54.529956 2261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:54.530901 2261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:54.530947 2261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.530987 2261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:54.531015 2261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.539382 2261 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:44087
I20260618 22:30:54.539410 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:44087 every 8 connection(s)
I20260618 22:30:54.540277 2261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:54.543395 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2261
I20260618 22:30:54.543514 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:30:54.543776 17358 external_mini_cluster.cc:1428] Setting key f3c157211dba963a0e84ff0ad4ae5453
Jun 18 22:30:54 dist-test-slave-jm5g krb5kdc[1886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:54.553570 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.541980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:55355 (local address 127.16.243.190:43939)
0618 22:30:54.542139 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.542145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.542992 (+ 847us) server_negotiation.cc:407] Connection header received
0618 22:30:54.544431 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.544455 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.544510 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.544587 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:54.545885 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.546423 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.547527 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.547744 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.547820 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.550017 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:54.550020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:54.550021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:54.550045 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:54.551429 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.552020 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.552023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.552025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.552108 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.552471 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.552473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.552474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.552660 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:54.552732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.553328 (+ 596us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.553434 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:30:54.554407 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.542382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:43939 (local address 127.16.243.131:55355)
0618 22:30:54.542850 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.542893 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.544238 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.544807 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.544814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.545221 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:54.545763 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.545780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.546517 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.546519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.547428 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.547437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.547855 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.547858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.547993 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.548495 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:54.548510 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:54.549875 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:54.551556 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.551561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.551575 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.551894 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.552220 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:54.552224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:54.552226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:54.552387 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:54.552745 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:54.552750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:54.553066 (+ 316us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.553259 (+ 193us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.553562 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":154,"threads_started":1}
I20260618 22:30:54.555546 2390 heartbeater.cc:347] Connected to a master server at 127.16.243.190:43939
I20260618 22:30:54.555769 2390 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:54.556344 2390 heartbeater.cc:511] Master 127.16.243.190:43939 requested a full tablet report, sending...
I20260618 22:30:54.557394 1928 ts_manager.cc:194] Registered new tserver with Master: ef170f1624e44964b353db76b0180826 (127.16.243.131:44087)
I20260618 22:30:54.557964 1928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:55355
I20260618 22:30:54.569415 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:30:54.584059 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.16.243.190:43939)
0618 22:30:54.571450 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.571454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.571644 (+ 190us) server_negotiation.cc:407] Connection header received
0618 22:30:54.571765 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.571768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.571812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.571874 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:54.572552 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.573074 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.574001 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.574217 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.574301 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.581096 (+ 6795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:54.581099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:54.581101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:54.581134 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:54.582264 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.582785 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.582787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.582789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.582875 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:54.583155 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:54.583159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:54.583161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:54.583404 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:54.583526 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.583779 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.583901 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:30:54.587155 1928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:30:54.590296 1928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:30:54.604470 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.598697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44087 (local address 127.0.0.1:37816)
0618 22:30:54.599114 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.599419 (+ 305us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.599544 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.600332 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.600338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.600360 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:54.600498 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.600504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.601666 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.601670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.602867 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.602878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.603946 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.603950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.604050 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.604097 (+ 47us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.604196 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.604318 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"spinlock_wait_cycles":8576,"thread_start_us":140,"threads_started":1}
I20260618 22:30:54.604475 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.598775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.16.243.131:44087)
0618 22:30:54.599521 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.599534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.599561 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:30:54.599750 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.599757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.599983 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.600157 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:54.600607 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.601531 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.602997 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.603847 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.603947 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.604084 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.604234 (+ 150us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.604324 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":90,"threads_started":1}
I20260618 22:30:54.605207 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.599382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33457 (local address 127.0.0.1:56064)
0618 22:30:54.600361 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.600373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.600458 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.600888 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.600891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.600908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:54.601002 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.601008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.602367 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.602370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.603502 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.603510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.604830 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.604834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.604927 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.604951 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.605009 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.605074 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":64,"thread_start_us":104,"threads_started":1}
I20260618 22:30:54.605468 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.598714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36983 (local address 127.0.0.1:51808)
0618 22:30:54.599706 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.599719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.599810 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.600505 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.600508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.600527 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:54.600636 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.600642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.602340 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.602344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.603634 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.603644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.604972 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.604976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.605063 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.605094 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.605182 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.605244 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260618 22:30:54.605610 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.599810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.16.243.129:33457)
0618 22:30:54.600230 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.600244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.600459 (+ 215us) server_negotiation.cc:407] Connection header received
0618 22:30:54.600566 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.600571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.600745 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.600884 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:54.601112 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.602243 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.603614 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.604524 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.604658 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.604773 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.605379 (+ 606us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.605463 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":91,"threads_started":1}
I20260618 22:30:54.605789 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.599572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.16.243.130:36983)
0618 22:30:54.600047 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.600060 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.600082 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:30:54.600154 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.600160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.600355 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.600778 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:54.600922 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.602194 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.603748 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.604524 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.604982 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.605112 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.605210 (+ 98us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.605465 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":117,"threads_started":1}
I20260618 22:30:54.607641 2324 tablet_service.cc:1467] Processing CreateTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 (DEFAULT_TABLE table=test-table [id=bb231cd64c2c4691b488f987becfef0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:54.607942 2190 tablet_service.cc:1467] Processing CreateTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 (DEFAULT_TABLE table=test-table [id=bb231cd64c2c4691b488f987becfef0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:54.608259 2056 tablet_service.cc:1467] Processing CreateTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 (DEFAULT_TABLE table=test-table [id=bb231cd64c2c4691b488f987becfef0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:54.608784 2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b39d6d73134c4084bb3b9cd6b849f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:54.608826 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b39d6d73134c4084bb3b9cd6b849f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:54.609150 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b39d6d73134c4084bb3b9cd6b849f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:54.614825 2407 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Bootstrap starting.
I20260618 22:30:54.616426 2409 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Bootstrap starting.
I20260618 22:30:54.616709 2408 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: Bootstrap starting.
I20260618 22:30:54.617479 2407 tablet_bootstrap.cc:654] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:54.618160 2409 tablet_bootstrap.cc:654] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:54.618763 2408 tablet_bootstrap.cc:654] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:54.618793 2407 log.cc:824] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:54.619234 2409 log.cc:824] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:54.619510 2408 log.cc:824] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:54.620869 2407 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: No bootstrap required, opened a new log
I20260618 22:30:54.621062 2407 ts_tablet_manager.cc:1397] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:30:54.621153 2409 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: No bootstrap required, opened a new log
I20260618 22:30:54.621330 2409 ts_tablet_manager.cc:1397] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:30:54.621366 2408 tablet_bootstrap.cc:492] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: No bootstrap required, opened a new log
I20260618 22:30:54.621505 2408 ts_tablet_manager.cc:1397] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:30:54.624035 2407 raft_consensus.cc:348] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.624260 2407 raft_consensus.cc:374] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:54.624312 2407 raft_consensus.cc:729] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc878032a824f5789b948f5dd40a67e, State: Initialized, Role: FOLLOWER
I20260618 22:30:54.624723 2407 consensus_queue.cc:260] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.625097 2409 raft_consensus.cc:348] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.625339 2409 raft_consensus.cc:374] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:54.625383 2409 raft_consensus.cc:729] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef170f1624e44964b353db76b0180826, State: Initialized, Role: FOLLOWER
I20260618 22:30:54.625676 2407 ts_tablet_manager.cc:1428] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:30:54.625550 2408 raft_consensus.cc:348] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.625761 2408 raft_consensus.cc:374] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:54.625813 2408 raft_consensus.cc:729] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fffd52e9cf4e6a95c82369c3857d21, State: Initialized, Role: FOLLOWER
I20260618 22:30:54.625831 2409 consensus_queue.cc:260] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.626013 2256 heartbeater.cc:503] Master 127.16.243.190:43939 was elected leader, sending a full tablet report...
I20260618 22:30:54.626256 2408 consensus_queue.cc:260] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.626752 2409 ts_tablet_manager.cc:1428] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260618 22:30:54.626909 2122 heartbeater.cc:503] Master 127.16.243.190:43939 was elected leader, sending a full tablet report...
I20260618 22:30:54.627079 2390 heartbeater.cc:503] Master 127.16.243.190:43939 was elected leader, sending a full tablet report...
I20260618 22:30:54.627727 2408 ts_tablet_manager.cc:1428] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260618 22:30:54.711122 2415 raft_consensus.cc:482] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:54.711321 2415 raft_consensus.cc:504] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.712630 2415 leader_election.cc:288] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef170f1624e44964b353db76b0180826 (127.16.243.131:44087), 6fc878032a824f5789b948f5dd40a67e (127.16.243.130:36983)
I20260618 22:30:54.716280 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.713011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:51083 (local address 127.16.243.131:44087)
0618 22:30:54.713138 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.713142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.713280 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:30:54.713395 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.713398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.713443 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.713506 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:54.713765 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.714261 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715380 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.715773 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715879 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.715935 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.716093 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.716160 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:30:54.716305 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.712869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:44087 (local address 127.16.243.129:51083)
0618 22:30:54.713191 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.713203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.713311 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.713519 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.713521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.713536 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:54.713670 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.713676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.714354 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.714358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.715266 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.715275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.715881 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.715971 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.716001 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.716065 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.716136 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":19,"thread_start_us":112,"threads_started":1}
I20260618 22:30:54.716360 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.713000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:33207 (local address 127.16.243.130:36983)
0618 22:30:54.713248 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.713252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.713438 (+ 186us) server_negotiation.cc:407] Connection header received
0618 22:30:54.713558 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.713561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.713608 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.713668 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:54.713921 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.714396 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715475 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.715866 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715969 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.716018 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.716190 (+ 172us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.716257 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260618 22:30:54.716372 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.713013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36983 (local address 127.16.243.129:33207)
0618 22:30:54.713351 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.713361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:54.713455 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:54.713689 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:54.713692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:54.713708 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:54.713817 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.713824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.714505 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.714508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.715357 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:54.715366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.715970 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.715974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:54.716078 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.716099 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:30:54.716162 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:30:54.716228 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":15,"thread_start_us":119,"threads_started":1}
I20260618 22:30:54.716987 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b39d6d73134c4084bb3b9cd6b849f0" candidate_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef170f1624e44964b353db76b0180826" is_pre_election: true
I20260618 22:30:54.717278 2344 raft_consensus.cc:2454] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5fffd52e9cf4e6a95c82369c3857d21 in term 0.
I20260618 22:30:54.717737 2010 leader_election.cc:302] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5fffd52e9cf4e6a95c82369c3857d21, ef170f1624e44964b353db76b0180826; no voters:
I20260618 22:30:54.717788 2210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b39d6d73134c4084bb3b9cd6b849f0" candidate_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc878032a824f5789b948f5dd40a67e" is_pre_election: true
I20260618 22:30:54.718032 2415 raft_consensus.cc:2781] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:30:54.718060 2210 raft_consensus.cc:2454] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5fffd52e9cf4e6a95c82369c3857d21 in term 0.
I20260618 22:30:54.718119 2415 raft_consensus.cc:482] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:54.718151 2415 raft_consensus.cc:3037] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:54.719211 2415 raft_consensus.cc:504] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.719519 2415 leader_election.cc:288] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [CANDIDATE]: Term 1 election: Requested vote from peers ef170f1624e44964b353db76b0180826 (127.16.243.131:44087), 6fc878032a824f5789b948f5dd40a67e (127.16.243.130:36983)
I20260618 22:30:54.719817 2210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b39d6d73134c4084bb3b9cd6b849f0" candidate_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc878032a824f5789b948f5dd40a67e"
I20260618 22:30:54.719833 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b39d6d73134c4084bb3b9cd6b849f0" candidate_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef170f1624e44964b353db76b0180826"
I20260618 22:30:54.719915 2210 raft_consensus.cc:3037] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:54.719957 2344 raft_consensus.cc:3037] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:54.720882 2210 raft_consensus.cc:2454] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5fffd52e9cf4e6a95c82369c3857d21 in term 1.
I20260618 22:30:54.720984 2344 raft_consensus.cc:2454] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5fffd52e9cf4e6a95c82369c3857d21 in term 1.
I20260618 22:30:54.721197 2011 leader_election.cc:302] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc878032a824f5789b948f5dd40a67e, d5fffd52e9cf4e6a95c82369c3857d21; no voters:
I20260618 22:30:54.721338 2415 raft_consensus.cc:2781] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:54.721525 2415 raft_consensus.cc:686] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 1 LEADER]: Becoming Leader. State: Replica: d5fffd52e9cf4e6a95c82369c3857d21, State: Running, Role: LEADER
I20260618 22:30:54.721815 2415 consensus_queue.cc:237] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } }
I20260618 22:30:54.724869 1930 catalog_manager.cc:5495] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5fffd52e9cf4e6a95c82369c3857d21 (127.16.243.129). New cstate: current_term: 1 leader_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5fffd52e9cf4e6a95c82369c3857d21" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 33457 } health_report { overall_health: HEALTHY } } }
I20260618 22:30:54.736303 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.732824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.16.243.129:33457)
0618 22:30:54.732946 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.732950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.732964 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:30:54.733106 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.733109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.733166 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.733251 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:54.733483 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.733986 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.734877 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.735089 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.735175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.735451 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:54.735864 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:30:54.735982 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.736098 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.736174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:30:54.741714 2210 raft_consensus.cc:1261] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 1 FOLLOWER]: Refusing update from remote peer d5fffd52e9cf4e6a95c82369c3857d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:54.741714 2344 raft_consensus.cc:1261] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 1 FOLLOWER]: Refusing update from remote peer d5fffd52e9cf4e6a95c82369c3857d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:54.742336 2418 consensus_queue.cc:1035] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc878032a824f5789b948f5dd40a67e" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:54.742525 2415 consensus_queue.cc:1035] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef170f1624e44964b353db76b0180826" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:54.751730 2422 mvcc.cc:205] Tried to move back new op lower bound from 7298342317015588864 to 7298342316943048704. Current Snapshot: MvccSnapshot[applied={T|T < 7298342317015588864}]
I20260618 22:30:54.752034 2423 mvcc.cc:205] Tried to move back new op lower bound from 7298342317015588864 to 7298342316943048704. Current Snapshot: MvccSnapshot[applied={T|T < 7298342317015588864}]
I20260618 22:30:54.757905 1930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:54.758100 1930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:54.760581 1930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41dc3d6f5d1a43838a7c8f20777f965e: Sending DeleteTablet for 3 replicas of tablet c8b39d6d73134c4084bb3b9cd6b849f0
I20260618 22:30:54.761425 2056 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:54 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56064
I20260618 22:30:54.761396 2190 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:54 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:51808
I20260618 22:30:54.761457 2324 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b39d6d73134c4084bb3b9cd6b849f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:54 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:37816
I20260618 22:30:54.761922 2433 tablet_replica.cc:330] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21: stopping tablet replica
I20260618 22:30:54.762158 2435 tablet_replica.cc:330] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: stopping tablet replica
I20260618 22:30:54.762262 2433 raft_consensus.cc:2229] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:30:54.762447 2434 tablet_replica.cc:330] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: stopping tablet replica
I20260618 22:30:54.762748 2433 raft_consensus.cc:2258] T c8b39d6d73134c4084bb3b9cd6b849f0 P d5fffd52e9cf4e6a95c82369c3857d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:54.762871 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1993
I20260618 22:30:54.764456 2434 raft_consensus.cc:2229] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:54.764750 2434 raft_consensus.cc:2258] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:54.762789 2435 raft_consensus.cc:2229] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:54.765522 2435 raft_consensus.cc:2258] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:54.766160 2434 ts_tablet_manager.cc:1905] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:54.766745 2435 ts_tablet_manager.cc:1905] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:54.768793 2435 ts_tablet_manager.cc:1918] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:54.768855 2434 ts_tablet_manager.cc:1918] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:54.768886 2435 log.cc:1196] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/c8b39d6d73134c4084bb3b9cd6b849f0
I20260618 22:30:54.768940 2434 log.cc:1196] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/c8b39d6d73134c4084bb3b9cd6b849f0
I20260618 22:30:54.769210 2435 ts_tablet_manager.cc:1939] T c8b39d6d73134c4084bb3b9cd6b849f0 P ef170f1624e44964b353db76b0180826: Deleting consensus metadata
I20260618 22:30:54.769249 2434 ts_tablet_manager.cc:1939] T c8b39d6d73134c4084bb3b9cd6b849f0 P 6fc878032a824f5789b948f5dd40a67e: Deleting consensus metadata
W20260618 22:30:54.769325 1918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:33457 (error 108)
I20260618 22:30:54.769505 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2127
I20260618 22:30:54.770869 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.770454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:33457 (local address 127.0.0.1:56074)
0618 22:30:54.770735 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.770795 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
W20260618 22:30:54.771025 1918 catalog_manager.cc:4584] TS d5fffd52e9cf4e6a95c82369c3857d21 (127.16.243.129:33457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8b39d6d73134c4084bb3b9cd6b849f0: Network error: Client connection negotiation failed: client connection to 127.16.243.129:33457: connect: Connection refused (error 111)
I20260618 22:30:54.771490 1916 catalog_manager.cc:4841] TS ef170f1624e44964b353db76b0180826 (127.16.243.131:44087): tablet c8b39d6d73134c4084bb3b9cd6b849f0 (table test-table [id=bb231cd64c2c4691b488f987becfef0e]) successfully deleted
I20260618 22:30:54.774730 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2261
I20260618 22:30:54.775508 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.774858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36983 (local address 127.0.0.1:51822)
0618 22:30:54.775376 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:54.775431 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:36983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
I20260618 22:30:54.779244 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 1899
2026-06-18T22:30:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2187 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-18T22:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:30:54Z Disabled control of system clock
I20260618 22:30:54.806322 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33089
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:33089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:30:54.913101 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:54.913371 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:54.913405 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:54.917359 2440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:54.917425 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:54.917439 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:54.917455 2440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:54.917469 2440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:54.922178 2440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:33089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:54.923313 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:54.924639 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:54.932610 2445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.932603 2446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:54.932605 2448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:54.932768 2440 server_base.cc:1034] running on GCE node
I20260618 22:30:54.933413 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:54.934525 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:54.935709 2440 hybrid_clock.cc:648] HybridClock initialized: now 1781821854935687 us; error 41 us; skew 500 ppm
I20260618 22:30:54.937472 2440 webserver.cc:460] Webserver started at http://127.16.243.190:45983/ using document root <none> and password file <none>
I20260618 22:30:54.938019 2440 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:54.938083 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:54.938265 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:54.940311 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "c019f78b77d340ccb2112235eb8b2b1a"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "6f19c0e6337eda104fb69f4e1cd9944d"
server_key_iv: "3d24a66d518b4c844daa8b8739dc5ba7"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.940850 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "c019f78b77d340ccb2112235eb8b2b1a"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "6f19c0e6337eda104fb69f4e1cd9944d"
server_key_iv: "3d24a66d518b4c844daa8b8739dc5ba7"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.944815 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260618 22:30:54.947341 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:54.948625 2440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:54.948741 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "c019f78b77d340ccb2112235eb8b2b1a"
format_stamp: "Formatted at 2026-06-18 22:30:54 on dist-test-slave-jm5g"
server_key: "6f19c0e6337eda104fb69f4e1cd9944d"
server_key_iv: "3d24a66d518b4c844daa8b8739dc5ba7"
server_key_version: "encryptionkey@0"
I20260618 22:30:54.948845 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:54.971086 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:54.971801 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:54.971977 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:54.978798 2440 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:33089
I20260618 22:30:54.978850 2506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:33089 every 8 connection(s)
I20260618 22:30:54.979768 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:54.981642 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2440
I20260618 22:30:54.981779 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:30:54.982084 17358 external_mini_cluster.cc:1428] Setting key 4533eacc1954f03a659cb56436f3be67
I20260618 22:30:54.983421 2507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:54.989241 2507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: Bootstrap starting.
I20260618 22:30:54.991211 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:54.983546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.16.243.190:33089)
0618 22:30:54.984200 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:54.984215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:54.984249 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:30:54.984888 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:54.984904 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:54.985318 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:54.985627 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:54.986046 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.987064 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.987912 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:54.988386 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:54.988529 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:54.989429 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:54.989436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:54.989452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:54.989863 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:54.989919 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:54.989940 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:54.990029 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:54.990173 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 22:30:54.990436 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260618 22:30:54.991657 2507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:54.992450 2507 log.cc:824] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:54.994473 2507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: No bootstrap required, opened a new log
I20260618 22:30:54.997999 2507 raft_consensus.cc:348] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } }
I20260618 22:30:54.998265 2507 raft_consensus.cc:374] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:54.998322 2507 raft_consensus.cc:729] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c019f78b77d340ccb2112235eb8b2b1a, State: Initialized, Role: FOLLOWER
I20260618 22:30:54.998795 2507 consensus_queue.cc:260] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } }
I20260618 22:30:54.998921 2507 raft_consensus.cc:388] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:54.998966 2507 raft_consensus.cc:482] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:54.999037 2507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:54.999939 2507 raft_consensus.cc:504] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } }
I20260618 22:30:55.000236 2507 leader_election.cc:302] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c019f78b77d340ccb2112235eb8b2b1a; no voters:
I20260618 22:30:55.000491 2507 leader_election.cc:288] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:30:55.000617 2512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:55.000818 2512 raft_consensus.cc:686] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [term 1 LEADER]: Becoming Leader. State: Replica: c019f78b77d340ccb2112235eb8b2b1a, State: Running, Role: LEADER
I20260618 22:30:55.001123 2512 consensus_queue.cc:237] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } }
I20260618 22:30:55.001660 2507 sys_catalog.cc:564] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:55.002722 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c019f78b77d340ccb2112235eb8b2b1a. Latest consensus state: current_term: 1 leader_uuid: "c019f78b77d340ccb2112235eb8b2b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } } }
I20260618 22:30:55.002708 2513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c019f78b77d340ccb2112235eb8b2b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c019f78b77d340ccb2112235eb8b2b1a" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33089 } } }
I20260618 22:30:55.002856 2513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:55.003270 2521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:55.002854 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:55.006361 2521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:55.011900 2521 catalog_manager.cc:1347] Generated new cluster ID: 031ea3e22664485ba5ece92dabbc667c
I20260618 22:30:55.011984 2521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:55.032579 2521 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:30:55.033573 2521 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:55.038831 2521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: Generated new TSK 0
I20260618 22:30:55.039508 2521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:30:55.045740 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--builtin_ntp_servers=127.16.243.148:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:30:55.153172 2531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:55.153453 2531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:55.153492 2531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:55.157508 2531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:55.157570 2531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:55.157629 2531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:55.162288 2531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:55.163378 2531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:55.164820 2531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:55.173915 2537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:55.174014 2536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:55.174052 2531 server_base.cc:1034] running on GCE node
W20260618 22:30:55.174124 2539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:55.174621 2531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:55.175248 2531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:55.176400 2531 hybrid_clock.cc:648] HybridClock initialized: now 1781821855176379 us; error 33 us; skew 500 ppm
I20260618 22:30:55.178126 2531 webserver.cc:460] Webserver started at http://127.16.243.129:37341/ using document root <none> and password file <none>
I20260618 22:30:55.178712 2531 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:55.178781 2531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:55.178969 2531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:55.180961 2531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "756eaa96e80b46b090406397c5fb1718"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "e9e9f0a1eb1d3c76ba880ddc5e3f2465"
server_key_iv: "5b0b6a3b4d5879633854a18265db3409"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.181495 2531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "756eaa96e80b46b090406397c5fb1718"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "e9e9f0a1eb1d3c76ba880ddc5e3f2465"
server_key_iv: "5b0b6a3b4d5879633854a18265db3409"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.185148 2531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:30:55.187625 2545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.188845 2531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:55.188958 2531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "756eaa96e80b46b090406397c5fb1718"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "e9e9f0a1eb1d3c76ba880ddc5e3f2465"
server_key_iv: "5b0b6a3b4d5879633854a18265db3409"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.189059 2531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:55.211280 2531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:55.212010 2531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:55.212193 2531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:55.212790 2531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:55.213752 2531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:55.213799 2531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.213852 2531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:55.213881 2531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.222419 2531 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:35139
I20260618 22:30:55.222450 2658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:35139 every 8 connection(s)
I20260618 22:30:55.223395 2531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:55.231772 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2531
I20260618 22:30:55.231930 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:30:55.232216 17358 external_mini_cluster.cc:1428] Setting key c3c3da8bc137165c90a227f674150e4f
I20260618 22:30:55.232750 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.225200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37749 (local address 127.16.243.190:33089)
0618 22:30:55.225320 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.225324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.226463 (+ 1139us) server_negotiation.cc:407] Connection header received
0618 22:30:55.227889 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.227892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.227960 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.228039 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:55.228942 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.229491 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.230678 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.230934 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.231027 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.232225 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:55.232228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:55.232230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:55.232243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:55.232263 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:55.232271 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:55.232331 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.232525 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.232603 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:30:55.233873 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.225618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33089 (local address 127.16.243.129:37749)
0618 22:30:55.226299 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.226343 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.227651 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.228208 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.228215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.228304 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:55.228791 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.228803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.229611 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.229614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.230522 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.230531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.231080 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.231083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.231231 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.231927 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:55.231929 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:55.231986 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:55.232061 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:55.232368 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:55.232379 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:55.232386 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.232542 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.232925 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260618 22:30:55.234232 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--builtin_ntp_servers=127.16.243.148:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:30:55.235108 2659 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33089
I20260618 22:30:55.235332 2659 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:55.235850 2659 heartbeater.cc:511] Master 127.16.243.190:33089 requested a full tablet report, sending...
I20260618 22:30:55.237319 2471 ts_manager.cc:194] Registered new tserver with Master: 756eaa96e80b46b090406397c5fb1718 (127.16.243.129:35139)
I20260618 22:30:55.238448 2471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.129:37749
W20260618 22:30:55.342113 2662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:55.342363 2662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:55.342399 2662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:55.346342 2662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:55.346401 2662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:55.346459 2662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:55.351140 2662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:55.352190 2662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:55.353551 2662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:55.362789 2670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:55.362821 2667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:55.362804 2668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:55.362804 2662 server_base.cc:1034] running on GCE node
I20260618 22:30:55.363400 2662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:55.364042 2662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:55.365198 2662 hybrid_clock.cc:648] HybridClock initialized: now 1781821855365180 us; error 31 us; skew 500 ppm
I20260618 22:30:55.367038 2662 webserver.cc:460] Webserver started at http://127.16.243.130:36839/ using document root <none> and password file <none>
I20260618 22:30:55.367588 2662 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:55.367654 2662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:55.367854 2662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:55.369829 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "98263f8fde254bc382e722c0f38f924b"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "16030a10dbdfd303dbe76b59cb756b2b"
server_key_iv: "424a3a033965628f92fed0310dceab2e"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.370335 2662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "98263f8fde254bc382e722c0f38f924b"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "16030a10dbdfd303dbe76b59cb756b2b"
server_key_iv: "424a3a033965628f92fed0310dceab2e"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.373716 2662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:30:55.375993 2676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.377120 2662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:30:55.377220 2662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "98263f8fde254bc382e722c0f38f924b"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "16030a10dbdfd303dbe76b59cb756b2b"
server_key_iv: "424a3a033965628f92fed0310dceab2e"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.377307 2662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:55.399446 2662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:55.400177 2662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:55.400345 2662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:55.400928 2662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:55.401886 2662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:55.401932 2662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.401983 2662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:55.402010 2662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.410266 2662 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:45365
I20260618 22:30:55.410305 2789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:45365 every 8 connection(s)
I20260618 22:30:55.411165 2662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:55.419708 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.412894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:33211 (local address 127.16.243.190:33089)
0618 22:30:55.413066 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.413071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.414100 (+ 1029us) server_negotiation.cc:407] Connection header received
0618 22:30:55.415472 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.415475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.415523 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.415600 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:55.416464 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.416980 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.418041 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.418249 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.418320 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.419327 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:55.419330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:55.419332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:55.419345 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:55.419364 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:55.419372 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:55.419427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.419534 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.419590 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:30:55.420454 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2662
I20260618 22:30:55.420519 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.413329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33089 (local address 127.16.243.130:33211)
0618 22:30:55.413959 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.413998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.415289 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.415756 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.415763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.415854 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:55.416342 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.416354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.417081 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.417084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.417936 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.417945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.418365 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.418367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.418505 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.419081 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:55.419083 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:55.419124 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:55.419189 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:55.419431 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:55.419435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:55.419439 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.419527 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.419746 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":116,"threads_started":1}
I20260618 22:30:55.420558 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:30:55.420821 17358 external_mini_cluster.cc:1428] Setting key 3c29203af1f5f929f1cd4173e15f4101
I20260618 22:30:55.421622 2790 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33089
I20260618 22:30:55.421844 2790 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:55.422322 2790 heartbeater.cc:511] Master 127.16.243.190:33089 requested a full tablet report, sending...
I20260618 22:30:55.422581 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--builtin_ntp_servers=127.16.243.148:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:30:55.423368 2471 ts_manager.cc:194] Registered new tserver with Master: 98263f8fde254bc382e722c0f38f924b (127.16.243.130:45365)
I20260618 22:30:55.423916 2471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.130:33211
W20260618 22:30:55.531888 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:55.532159 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:55.532198 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:55.536310 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:55.536373 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:55.536432 2793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:55.541546 2793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:55.542682 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:55.544075 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:55.553504 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:55.553463 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:55.553686 2793 server_base.cc:1034] running on GCE node
W20260618 22:30:55.553505 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:55.554248 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:55.554924 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:55.556070 2793 hybrid_clock.cc:648] HybridClock initialized: now 1781821855556052 us; error 33 us; skew 500 ppm
I20260618 22:30:55.557883 2793 webserver.cc:460] Webserver started at http://127.16.243.131:33775/ using document root <none> and password file <none>
I20260618 22:30:55.558440 2793 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:55.558502 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:55.558732 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:55.560742 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "4babd63ff03a4c21bf18cabde3ae5bcc"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "8bf6450a0a8dbd1081d50c81e7ec2c55"
server_key_iv: "eb1430cca4f7bb0699ec4d69aafd7c1b"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.561260 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "4babd63ff03a4c21bf18cabde3ae5bcc"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "8bf6450a0a8dbd1081d50c81e7ec2c55"
server_key_iv: "eb1430cca4f7bb0699ec4d69aafd7c1b"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.564847 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:30:55.567211 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.568446 2793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:55.568548 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "4babd63ff03a4c21bf18cabde3ae5bcc"
format_stamp: "Formatted at 2026-06-18 22:30:55 on dist-test-slave-jm5g"
server_key: "8bf6450a0a8dbd1081d50c81e7ec2c55"
server_key_iv: "eb1430cca4f7bb0699ec4d69aafd7c1b"
server_key_version: "encryptionkey@0"
I20260618 22:30:55.568652 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:55.587249 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:55.587944 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:55.588114 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:55.588693 2793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:55.589612 2793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:55.589656 2793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.589708 2793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:55.589737 2793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:55.598194 2793 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:46411
I20260618 22:30:55.598227 2920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:46411 every 8 connection(s)
I20260618 22:30:55.599119 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:55.607842 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.600877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:43889 (local address 127.16.243.190:33089)
0618 22:30:55.601015 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.601019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.602119 (+ 1100us) server_negotiation.cc:407] Connection header received
0618 22:30:55.603640 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.603642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.603692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.603765 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:55.604652 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.605159 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.606212 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.606413 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.606477 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.607474 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:55.607477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:55.607479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:55.607492 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:55.607509 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:55.607517 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:55.607570 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.607672 (+ 102us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.607723 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:30:55.608680 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2793
I20260618 22:30:55.608680 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.601265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:33089 (local address 127.16.243.131:43889)
0618 22:30:55.601908 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.602027 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.603435 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.603925 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.603932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.604023 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:55.604542 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.604555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.605244 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.605246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.606124 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.606132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.606527 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.606529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.606683 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.607233 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:55.607235 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:55.607275 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:55.607340 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:55.607575 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:55.607580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:55.607584 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.607667 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.607883 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260618 22:30:55.608774 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:30:55.609007 17358 external_mini_cluster.cc:1428] Setting key a1dc6f2020a7973aabff26abcdc6067f
I20260618 22:30:55.609797 2921 heartbeater.cc:347] Connected to a master server at 127.16.243.190:33089
I20260618 22:30:55.610033 2921 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:55.610534 2921 heartbeater.cc:511] Master 127.16.243.190:33089 requested a full tablet report, sending...
I20260618 22:30:55.611527 2471 ts_manager.cc:194] Registered new tserver with Master: 4babd63ff03a4c21bf18cabde3ae5bcc (127.16.243.131:46411)
I20260618 22:30:55.612063 2471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.131:43889
I20260618 22:30:55.622473 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:30:55.627452 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.624339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.16.243.190:33089)
0618 22:30:55.624447 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.624450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.624608 (+ 158us) server_negotiation.cc:407] Connection header received
0618 22:30:55.624709 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.624711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.624751 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.624810 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:55.624990 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.625481 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.626246 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.626438 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.626502 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.627153 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:55.627156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:55.627158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:55.627169 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:55.627187 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:55.627195 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:55.627244 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.627299 (+ 55us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.627344 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:30:55.629909 2471 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:30:55.632815 2471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:30:55.646580 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35139 (local address 127.0.0.1:52982)
0618 22:30:55.642013 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.642025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.642114 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.642804 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.642807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.642839 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.643044 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.643050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.644003 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.644006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.645130 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.645139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.646029 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.646032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.646139 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.646175 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.646280 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.646400 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":103,"spinlock_wait_cycles":11648,"thread_start_us":84,"threads_started":1}
I20260618 22:30:55.646597 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.16.243.129:35139)
0618 22:30:55.642265 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.642277 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.642303 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:30:55.642378 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.642383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.642651 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.642903 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.643173 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.643903 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.645242 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.645935 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.646031 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.646214 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.646304 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.646399 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":103,"threads_started":1}
I20260618 22:30:55.647145 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:46411 (local address 127.0.0.1:50584)
0618 22:30:55.641371 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.641423 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.641563 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.642804 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.642807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.642839 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.643009 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.643016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.644527 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.644530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.645782 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.645793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.646785 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.646789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.646874 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.646896 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.646940 (+ 44us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.647006 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":70,"spinlock_wait_cycles":14848,"thread_start_us":92,"threads_started":1}
I20260618 22:30:55.647265 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45365 (local address 127.0.0.1:48336)
0618 22:30:55.642015 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.642025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.642118 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.642784 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.642790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.642811 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.642998 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.643006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.644525 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.644528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.645666 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.645675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.646931 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.646935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.647030 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.647050 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.647103 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.647160 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":80,"spinlock_wait_cycles":8064,"thread_start_us":105,"threads_started":1}
I20260618 22:30:55.647840 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.16.243.131:46411)
0618 22:30:55.642419 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.642428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.642447 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:30:55.642508 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.642513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.642673 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.643151 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.643280 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.644415 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.645907 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.646670 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.647343 (+ 673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.647540 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.647591 (+ 51us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.647689 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":168,"threads_started":1}
I20260618 22:30:55.647876 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.641701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.16.243.130:45365)
0618 22:30:55.642361 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.642373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.642392 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:30:55.642451 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.642454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.642649 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.642784 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.643143 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.644377 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.645783 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.646586 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.647477 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.647621 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.647660 (+ 39us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.647736 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":112,"threads_started":1}
I20260618 22:30:55.649616 2593 tablet_service.cc:1467] Processing CreateTablet for tablet 106f74fd61a6402092b011790d63f10f (DEFAULT_TABLE table=test-table [id=7c64942272dd42d79192851369e3bc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:55.650188 2855 tablet_service.cc:1467] Processing CreateTablet for tablet 106f74fd61a6402092b011790d63f10f (DEFAULT_TABLE table=test-table [id=7c64942272dd42d79192851369e3bc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:55.650774 2593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106f74fd61a6402092b011790d63f10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:55.651063 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106f74fd61a6402092b011790d63f10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:55.650862 2724 tablet_service.cc:1467] Processing CreateTablet for tablet 106f74fd61a6402092b011790d63f10f (DEFAULT_TABLE table=test-table [id=7c64942272dd42d79192851369e3bc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:55.651777 2724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106f74fd61a6402092b011790d63f10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:55.658183 2938 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Bootstrap starting.
I20260618 22:30:55.658512 2939 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Bootstrap starting.
I20260618 22:30:55.658818 2940 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Bootstrap starting.
I20260618 22:30:55.660274 2939 tablet_bootstrap.cc:654] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:55.660446 2938 tablet_bootstrap.cc:654] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:55.660558 2940 tablet_bootstrap.cc:654] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:55.660996 2939 log.cc:824] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:55.661262 2938 log.cc:824] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:55.661262 2940 log.cc:824] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:55.662966 2940 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: No bootstrap required, opened a new log
I20260618 22:30:55.663049 2938 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: No bootstrap required, opened a new log
I20260618 22:30:55.663175 2940 ts_tablet_manager.cc:1397] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260618 22:30:55.663242 2938 ts_tablet_manager.cc:1397] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:30:55.663565 2939 tablet_bootstrap.cc:492] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: No bootstrap required, opened a new log
I20260618 22:30:55.663720 2939 ts_tablet_manager.cc:1397] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:30:55.666335 2940 raft_consensus.cc:348] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.666574 2940 raft_consensus.cc:374] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:55.666622 2940 raft_consensus.cc:729] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98263f8fde254bc382e722c0f38f924b, State: Initialized, Role: FOLLOWER
I20260618 22:30:55.667024 2940 consensus_queue.cc:260] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.667285 2939 raft_consensus.cc:348] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.667285 2938 raft_consensus.cc:348] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.667536 2939 raft_consensus.cc:374] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:55.667549 2938 raft_consensus.cc:374] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:55.667598 2938 raft_consensus.cc:729] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4babd63ff03a4c21bf18cabde3ae5bcc, State: Initialized, Role: FOLLOWER
I20260618 22:30:55.667598 2939 raft_consensus.cc:729] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756eaa96e80b46b090406397c5fb1718, State: Initialized, Role: FOLLOWER
I20260618 22:30:55.667706 2790 heartbeater.cc:503] Master 127.16.243.190:33089 was elected leader, sending a full tablet report...
I20260618 22:30:55.667986 2940 ts_tablet_manager.cc:1428] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:30:55.668008 2938 consensus_queue.cc:260] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.668008 2939 consensus_queue.cc:260] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.668849 2659 heartbeater.cc:503] Master 127.16.243.190:33089 was elected leader, sending a full tablet report...
I20260618 22:30:55.668857 2921 heartbeater.cc:503] Master 127.16.243.190:33089 was elected leader, sending a full tablet report...
I20260618 22:30:55.668946 2938 ts_tablet_manager.cc:1428] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:30:55.668946 2939 ts_tablet_manager.cc:1428] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:30:55.706408 2946 raft_consensus.cc:482] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:55.706624 2946 raft_consensus.cc:504] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.707695 2946 leader_election.cc:288] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 756eaa96e80b46b090406397c5fb1718 (127.16.243.129:35139), 98263f8fde254bc382e722c0f38f924b (127.16.243.130:45365)
I20260618 22:30:55.709440 2944 raft_consensus.cc:482] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:55.709620 2944 raft_consensus.cc:504] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.710768 2944 leader_election.cc:288] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4babd63ff03a4c21bf18cabde3ae5bcc (127.16.243.131:46411), 756eaa96e80b46b090406397c5fb1718 (127.16.243.129:35139)
I20260618 22:30:55.711566 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.708030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:43089 (local address 127.16.243.130:45365)
0618 22:30:55.708152 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.708156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.708406 (+ 250us) server_negotiation.cc:407] Connection header received
0618 22:30:55.708518 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.708521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.708566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.708626 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.708876 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.709412 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.710507 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.711055 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.711158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.711203 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.711379 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.711449 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:30:55.711607 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.707883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35139 (local address 127.16.243.131:40883)
0618 22:30:55.708073 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.708086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.708179 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.708415 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.708418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.708434 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.708535 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.708541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.709230 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.709233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.710316 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.710325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.711272 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.711275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.711358 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.711376 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.711430 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.711491 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1}
I20260618 22:30:55.711978 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.707934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45365 (local address 127.16.243.131:43089)
0618 22:30:55.708312 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.708324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.708419 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.708642 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.708645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.708660 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.708776 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.708782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.709506 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.709509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.710345 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.710354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.711156 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.711160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.711249 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.711271 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.711785 (+ 514us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.711854 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260618 22:30:55.712007 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.707965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:40883 (local address 127.16.243.129:35139)
0618 22:30:55.708085 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.708090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.708158 (+ 68us) server_negotiation.cc:407] Connection header received
0618 22:30:55.708283 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.708286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.708331 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.708392 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.708640 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.709131 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.710447 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.710949 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.711070 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.711121 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.711832 (+ 711us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.711905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:30:55.712476 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756eaa96e80b46b090406397c5fb1718" is_pre_election: true
I20260618 22:30:55.712754 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98263f8fde254bc382e722c0f38f924b" is_pre_election: true
I20260618 22:30:55.712786 2613 raft_consensus.cc:2454] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4babd63ff03a4c21bf18cabde3ae5bcc in term 0.
I20260618 22:30:55.712941 2744 raft_consensus.cc:2454] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4babd63ff03a4c21bf18cabde3ae5bcc in term 0.
I20260618 22:30:55.713290 2809 leader_election.cc:302] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4babd63ff03a4c21bf18cabde3ae5bcc, 756eaa96e80b46b090406397c5fb1718; no voters:
I20260618 22:30:55.713660 2946 raft_consensus.cc:2781] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:30:55.713760 2946 raft_consensus.cc:482] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:55.713809 2946 raft_consensus.cc:3037] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:55.714820 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.711058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39687 (local address 127.16.243.131:46411)
0618 22:30:55.711213 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.711218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.711539 (+ 321us) server_negotiation.cc:407] Connection header received
0618 22:30:55.711579 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.711581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.711620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.711676 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.711976 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.712588 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.713926 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.714303 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.714411 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.714452 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.714659 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.714716 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260618 22:30:55.714820 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.710926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:46411 (local address 127.16.243.130:39687)
0618 22:30:55.711310 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.711324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.711398 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.711759 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.711761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.711776 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.711864 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.711870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.712708 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.712711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.713791 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.713802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.714424 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.714428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.714524 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.714579 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.714647 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.714716 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260618 22:30:55.715067 2946 raft_consensus.cc:504] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.715301 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "98263f8fde254bc382e722c0f38f924b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" is_pre_election: true
I20260618 22:30:55.715409 2946 leader_election.cc:288] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [CANDIDATE]: Term 1 election: Requested vote from peers 756eaa96e80b46b090406397c5fb1718 (127.16.243.129:35139), 98263f8fde254bc382e722c0f38f924b (127.16.243.130:45365)
I20260618 22:30:55.715514 2875 raft_consensus.cc:2379] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98263f8fde254bc382e722c0f38f924b in current term 1: Already voted for candidate 4babd63ff03a4c21bf18cabde3ae5bcc in this term.
I20260618 22:30:55.715718 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756eaa96e80b46b090406397c5fb1718"
I20260618 22:30:55.715824 2613 raft_consensus.cc:3037] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:55.715979 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.711929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:51171 (local address 127.16.243.129:35139)
0618 22:30:55.712133 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.712136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.712148 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:30:55.712182 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.712185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.712222 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.712286 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:55.712740 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.713505 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.714840 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.715347 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.715432 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.715472 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.715798 (+ 326us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.715863 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260618 22:30:55.715940 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98263f8fde254bc382e722c0f38f924b"
I20260618 22:30:55.716039 2744 raft_consensus.cc:3037] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:55.716287 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.711152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35139 (local address 127.16.243.130:51171)
0618 22:30:55.711483 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.711494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:55.711579 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:55.712482 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:55.712484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:55.712501 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:55.712625 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.712631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.713627 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.713630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.714710 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:55.714720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.715574 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.715578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:55.715668 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.715689 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:30:55.716092 (+ 403us) client_negotiation.cc:240] Negotiation successful
0618 22:30:55.716165 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260618 22:30:55.716670 2612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106f74fd61a6402092b011790d63f10f" candidate_uuid: "98263f8fde254bc382e722c0f38f924b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756eaa96e80b46b090406397c5fb1718" is_pre_election: true
I20260618 22:30:55.717023 2613 raft_consensus.cc:2454] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4babd63ff03a4c21bf18cabde3ae5bcc in term 1.
I20260618 22:30:55.717128 2744 raft_consensus.cc:2454] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4babd63ff03a4c21bf18cabde3ae5bcc in term 1.
I20260618 22:30:55.717321 2809 leader_election.cc:302] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4babd63ff03a4c21bf18cabde3ae5bcc, 756eaa96e80b46b090406397c5fb1718; no voters:
I20260618 22:30:55.717375 2678 leader_election.cc:302] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98263f8fde254bc382e722c0f38f924b; no voters: 4babd63ff03a4c21bf18cabde3ae5bcc, 756eaa96e80b46b090406397c5fb1718
I20260618 22:30:55.717566 2946 raft_consensus.cc:2781] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:55.717785 2946 raft_consensus.cc:686] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 LEADER]: Becoming Leader. State: Replica: 4babd63ff03a4c21bf18cabde3ae5bcc, State: Running, Role: LEADER
I20260618 22:30:55.717809 2944 raft_consensus.cc:2726] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260618 22:30:55.718161 2946 consensus_queue.cc:237] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } }
I20260618 22:30:55.720973 2470 catalog_manager.cc:5495] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4babd63ff03a4c21bf18cabde3ae5bcc (127.16.243.131). New cstate: current_term: 1 leader_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4babd63ff03a4c21bf18cabde3ae5bcc" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 46411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 } health_report { overall_health: UNKNOWN } } }
W20260618 22:30:55.724545 2660 tablet.cc:2369] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:30:55.745077 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.741665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.16.243.131:46411)
0618 22:30:55.741804 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:55.741809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:55.741822 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:30:55.741897 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:55.741899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:55.741950 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:55.742038 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:55.742272 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.742806 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.743667 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:55.743881 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:55.743967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:55.744259 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:55.744689 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:30:55.744800 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:55.744860 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 22:30:55.744916 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:30:55.750638 2613 raft_consensus.cc:1261] T 106f74fd61a6402092b011790d63f10f P 756eaa96e80b46b090406397c5fb1718 [term 1 FOLLOWER]: Refusing update from remote peer 4babd63ff03a4c21bf18cabde3ae5bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:55.750689 2744 raft_consensus.cc:1261] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 1 FOLLOWER]: Refusing update from remote peer 4babd63ff03a4c21bf18cabde3ae5bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:55.751278 2950 consensus_queue.cc:1035] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "756eaa96e80b46b090406397c5fb1718" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:55.751446 2946 consensus_queue.cc:1035] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "98263f8fde254bc382e722c0f38f924b" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:55.760354 2954 mvcc.cc:205] Tried to move back new op lower bound from 7298342321147957248 to 7298342321024172032. Current Snapshot: MvccSnapshot[applied={T|T < 7298342321147957248}]
I20260618 22:30:55.766640 2469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:55.766824 2469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:55.769222 2469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c019f78b77d340ccb2112235eb8b2b1a: Sending DeleteTablet for 3 replicas of tablet 106f74fd61a6402092b011790d63f10f
I20260618 22:30:55.770066 2724 tablet_service.cc:1514] Processing DeleteTablet for tablet 106f74fd61a6402092b011790d63f10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:55 UTC) from {username='slave'} at 127.0.0.1:48336
I20260618 22:30:55.770062 2855 tablet_service.cc:1514] Processing DeleteTablet for tablet 106f74fd61a6402092b011790d63f10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:55 UTC) from {username='slave'} at 127.0.0.1:50584
I20260618 22:30:55.770166 2593 tablet_service.cc:1514] Processing DeleteTablet for tablet 106f74fd61a6402092b011790d63f10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:55 UTC) from {username='slave'} at 127.0.0.1:52982
I20260618 22:30:55.770632 2964 tablet_replica.cc:330] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: stopping tablet replica
I20260618 22:30:55.770941 2964 raft_consensus.cc:2229] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:30:55.771337 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2531
I20260618 22:30:55.771359 2964 raft_consensus.cc:2258] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:55.772915 2964 ts_tablet_manager.cc:1905] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:55.775672 2964 ts_tablet_manager.cc:1918] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:55.775758 2964 log.cc:1196] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/106f74fd61a6402092b011790d63f10f
I20260618 22:30:55.776001 2964 ts_tablet_manager.cc:1939] T 106f74fd61a6402092b011790d63f10f P 4babd63ff03a4c21bf18cabde3ae5bcc: Deleting consensus metadata
I20260618 22:30:55.777069 2963 tablet_replica.cc:330] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: stopping tablet replica
I20260618 22:30:55.777048 2456 catalog_manager.cc:4841] TS 4babd63ff03a4c21bf18cabde3ae5bcc (127.16.243.131:46411): tablet 106f74fd61a6402092b011790d63f10f (table test-table [id=7c64942272dd42d79192851369e3bc67]) successfully deleted
I20260618 22:30:55.777376 2963 raft_consensus.cc:2229] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:55.777678 2963 raft_consensus.cc:2258] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:55.779115 2963 ts_tablet_manager.cc:1905] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:30:55.781381 2456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:35139 (error 108)
I20260618 22:30:55.781733 2963 ts_tablet_manager.cc:1918] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:55.781829 2963 log.cc:1196] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/106f74fd61a6402092b011790d63f10f
I20260618 22:30:55.782116 2963 ts_tablet_manager.cc:1939] T 106f74fd61a6402092b011790d63f10f P 98263f8fde254bc382e722c0f38f924b: Deleting consensus metadata
I20260618 22:30:55.782513 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2662
I20260618 22:30:55.782608 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.782298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35139 (local address 127.0.0.1:52994)
0618 22:30:55.782416 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.782484 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:35139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:30:55.786710 2456 catalog_manager.cc:4584] TS 756eaa96e80b46b090406397c5fb1718 (127.16.243.129:35139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 106f74fd61a6402092b011790d63f10f: Network error: Client connection negotiation failed: client connection to 127.16.243.129:35139: connect: Connection refused (error 111)
I20260618 22:30:55.788218 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2793
I20260618 22:30:55.788463 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:55.788165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:45365 (local address 127.0.0.1:48352)
0618 22:30:55.788343 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:55.788388 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.130:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260618 22:30:55.792779 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2440
2026-06-18T22:30:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1012 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:30:55 dist-test-slave-jm5g krb5kdc[2969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:30:55 dist-test-slave-jm5g krb5kdc[2969](info): set up 2 sockets
Jun 18 22:30:55 dist-test-slave-jm5g krb5kdc[2969](info): commencing operation
krb5kdc: starting...
W20260618 22:30:56.750190 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:30:56 dist-test-slave-jm5g krb5kdc[2969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:30:56.862736 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:46617
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:46617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:56.970167 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:56.970461 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:56.970518 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:56.974591 2982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:56.974658 2982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:56.974684 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:56.974706 2982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:56.974735 2982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:56.980010 2982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:46617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:56.981035 2982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:56.982038 2982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:56.990190 2988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:56.990161 2987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:56.990263 2982 server_base.cc:1034] running on GCE node
W20260618 22:30:56.990155 2990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:56.990989 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:56.992075 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:56.993255 2982 hybrid_clock.cc:648] HybridClock initialized: now 1781821856993233 us; error 34 us; skew 500 ppm
Jun 18 22:30:56 dist-test-slave-jm5g krb5kdc[2969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821856, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:56.995922 2982 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:30:56.996762 2982 webserver.cc:460] Webserver started at http://127.16.243.190:44545/ using document root <none> and password file <none>
I20260618 22:30:56.997291 2982 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:56.997355 2982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:56.997565 2982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:56.999652 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "865b4ba15f074acdad5c4183dc331705"
format_stamp: "Formatted at 2026-06-18 22:30:56 on dist-test-slave-jm5g"
server_key: "a25f0e0d7d578763d74ad5a500e8ed71"
server_key_iv: "f2e0b37b69b8b4cf29d5251b87b22e64"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.000172 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "865b4ba15f074acdad5c4183dc331705"
format_stamp: "Formatted at 2026-06-18 22:30:56 on dist-test-slave-jm5g"
server_key: "a25f0e0d7d578763d74ad5a500e8ed71"
server_key_iv: "f2e0b37b69b8b4cf29d5251b87b22e64"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.003733 2982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:30:57.006094 2997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.007302 2982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:30:57.007421 2982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "865b4ba15f074acdad5c4183dc331705"
format_stamp: "Formatted at 2026-06-18 22:30:56 on dist-test-slave-jm5g"
server_key: "a25f0e0d7d578763d74ad5a500e8ed71"
server_key_iv: "f2e0b37b69b8b4cf29d5251b87b22e64"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.007522 2982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:57.027976 2982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:57.031140 2982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:57.031317 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:57.037961 2982 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:46617
I20260618 22:30:57.037976 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:46617 every 8 connection(s)
I20260618 22:30:57.039078 2982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:57.041823 3050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:57.047142 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: Bootstrap starting.
I20260618 22:30:57.048686 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 2982
I20260618 22:30:57.048836 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:30:57.049095 17358 external_mini_cluster.cc:1428] Setting key 88752427577dad49fd60ff8f2ac2c75b
I20260618 22:30:57.049566 3050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:57.050369 3050 log.cc:824] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:57.052192 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: No bootstrap required, opened a new log
I20260618 22:30:57.056126 3050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } }
I20260618 22:30:57.056365 3050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:57.056423 3050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865b4ba15f074acdad5c4183dc331705, State: Initialized, Role: FOLLOWER
I20260618 22:30:57.056926 3050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } }
I20260618 22:30:57.057085 3050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:57.057148 3050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:57.057243 3050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 0 FOLLOWER]: Advancing to term 1
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:57.058171 3050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } }
I20260618 22:30:57.058516 3050 leader_election.cc:302] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 865b4ba15f074acdad5c4183dc331705; no voters:
I20260618 22:30:57.058861 3050 leader_election.cc:288] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:30:57.059126 3055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:57.059374 3055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [term 1 LEADER]: Becoming Leader. State: Replica: 865b4ba15f074acdad5c4183dc331705, State: Running, Role: LEADER
I20260618 22:30:57.059754 3055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } }
I20260618 22:30:57.060062 3050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:57.061347 3057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865b4ba15f074acdad5c4183dc331705. Latest consensus state: current_term: 1 leader_uuid: "865b4ba15f074acdad5c4183dc331705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } } }
I20260618 22:30:57.061458 3057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:57.061878 3056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "865b4ba15f074acdad5c4183dc331705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865b4ba15f074acdad5c4183dc331705" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 46617 } } }
I20260618 22:30:57.062222 3066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:57.062760 3056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705 [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:57.063688 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.050227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.16.243.190:46617)
0618 22:30:57.050756 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.050777 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.050813 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:30:57.051476 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.051492 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.051806 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.052095 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:57.052957 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.053959 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.055190 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.055664 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.055787 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.058474 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:57.058479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:57.058492 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:57.058520 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:57.060503 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.060999 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.061005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.061011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.061167 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.061470 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.061473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.061475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.061997 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:57.062198 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.062652 (+ 454us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.062956 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":181,"threads_started":1}
I20260618 22:30:57.065330 3066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:57.070341 3066 catalog_manager.cc:1347] Generated new cluster ID: 2205784722df4401885d969b235fa408
I20260618 22:30:57.070403 3066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:57.084493 3066 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:30:57.085395 3066 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:57.097779 3066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: Generated new TSK 0
I20260618 22:30:57.098516 3066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:30:57.129886 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:57.237584 3076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:57.237883 3076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:57.237927 3076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:57.241933 3076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:57.241995 3076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:57.242054 3076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:57.247326 3076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:57.248322 3076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:57.249270 3076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:57.258291 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:57.258323 3081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:57.258299 3082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:57.258623 3076 server_base.cc:1034] running on GCE node
I20260618 22:30:57.258978 3076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:57.259572 3076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:57.260720 3076 hybrid_clock.cc:648] HybridClock initialized: now 1781821857260690 us; error 41 us; skew 500 ppm
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:57.263243 3076 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:30:57.264027 3076 webserver.cc:460] Webserver started at http://127.16.243.129:36025/ using document root <none> and password file <none>
I20260618 22:30:57.264518 3076 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:57.264571 3076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:57.264742 3076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:57.266781 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "70d7e01a78d841a8b9ba50fe37da21f8"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "e838ccf4b6cf30640ba535018019280c"
server_key_iv: "83e7c74d9d6e9b4635ca953770b8d8c8"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.267323 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "70d7e01a78d841a8b9ba50fe37da21f8"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "e838ccf4b6cf30640ba535018019280c"
server_key_iv: "83e7c74d9d6e9b4635ca953770b8d8c8"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.270720 3076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:30:57.272908 3091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.274053 3076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:57.274156 3076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "70d7e01a78d841a8b9ba50fe37da21f8"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "e838ccf4b6cf30640ba535018019280c"
server_key_iv: "83e7c74d9d6e9b4635ca953770b8d8c8"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.274251 3076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:57.298207 3076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:57.301095 3076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:57.301265 3076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:57.301817 3076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:57.302666 3076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:57.302712 3076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.302772 3076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:57.302802 3076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.310875 3076 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:39413
I20260618 22:30:57.311224 3204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:39413 every 8 connection(s)
I20260618 22:30:57.311827 3076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:57.315668 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3076
I20260618 22:30:57.315795 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:30:57.316084 17358 external_mini_cluster.cc:1428] Setting key c212e6de9ce51a4e218f1f2baa330226
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:30:57.325500 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.313473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:52275 (local address 127.16.243.190:46617)
0618 22:30:57.313607 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.313610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.314439 (+ 829us) server_negotiation.cc:407] Connection header received
0618 22:30:57.315281 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.315284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.315330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.315398 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:57.316845 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.317567 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.319120 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.319458 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.319561 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.321834 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:57.321837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:57.321839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:57.321864 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:57.323213 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.323771 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.323774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.323775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.323856 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.324292 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.324295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.324296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.324490 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:57.324576 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.325197 (+ 621us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.325326 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:30:57.326874 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.313847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:46617 (local address 127.16.243.129:52275)
0618 22:30:57.314288 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.314330 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.315103 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.315617 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.315627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.316080 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:57.316698 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.316718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.317686 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.317689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.319000 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.319012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.319593 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.319596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.319733 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.320304 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:57.320320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:57.321665 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:57.323323 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.323328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.323345 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.323681 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.324004 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.324007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.324009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.324212 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.324597 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:57.324604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:57.324926 (+ 322us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.325655 (+ 729us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.325954 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:30:57.328054 3205 heartbeater.cc:347] Connected to a master server at 127.16.243.190:46617
I20260618 22:30:57.328301 3205 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:57.328847 3205 heartbeater.cc:511] Master 127.16.243.190:46617 requested a full tablet report, sending...
I20260618 22:30:57.330328 3014 ts_manager.cc:194] Registered new tserver with Master: 70d7e01a78d841a8b9ba50fe37da21f8 (127.16.243.129:39413)
I20260618 22:30:57.331506 3014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:52275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:30:57.340957 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:57.445647 3210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:57.445902 3210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:57.445961 3210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:57.449923 3210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:57.449985 3210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:57.450065 3210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:57.455225 3210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:57.456178 3210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:57.457109 3210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:57.466382 3218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:57.466351 3215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:57.466362 3210 server_base.cc:1034] running on GCE node
W20260618 22:30:57.466393 3216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:57.467041 3210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:57.467653 3210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:57.468796 3210 hybrid_clock.cc:648] HybridClock initialized: now 1781821857468773 us; error 46 us; skew 500 ppm
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:57.471306 3210 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:30:57.472112 3210 webserver.cc:460] Webserver started at http://127.16.243.130:41195/ using document root <none> and password file <none>
I20260618 22:30:57.472626 3210 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:57.472687 3210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:57.472862 3210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:57.474817 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "d146d0a0012640d6aea8ff5ee2424be1"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "456172d5ac478f9630e8a5a1b780068f"
server_key_iv: "73743721770d313a7fe0cd7f79b6d320"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.475308 3210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "d146d0a0012640d6aea8ff5ee2424be1"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "456172d5ac478f9630e8a5a1b780068f"
server_key_iv: "73743721770d313a7fe0cd7f79b6d320"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.478861 3210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:30:57.481021 3225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.482141 3210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:30:57.482244 3210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "d146d0a0012640d6aea8ff5ee2424be1"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "456172d5ac478f9630e8a5a1b780068f"
server_key_iv: "73743721770d313a7fe0cd7f79b6d320"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.482326 3210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:57.508772 3210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:57.511763 3210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:57.511934 3210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:57.512521 3210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:57.513325 3210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:57.513370 3210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.513420 3210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:57.513448 3210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.521728 3210 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:35563
I20260618 22:30:57.521759 3338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:35563 every 8 connection(s)
I20260618 22:30:57.522608 3210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:57.526859 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3210
I20260618 22:30:57.526983 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:30:57.527369 17358 external_mini_cluster.cc:1428] Setting key 6f4b58ff866da5bc1ac28f8b9daa2ca5
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:30:57.538817 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.524331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:54633 (local address 127.16.243.190:46617)
0618 22:30:57.524478 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.524483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.525633 (+ 1150us) server_negotiation.cc:407] Connection header received
0618 22:30:57.527424 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.527429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.527492 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.527594 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:57.529083 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.530022 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.531392 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.531711 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.531806 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.534778 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:57.534782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:57.534785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:57.534815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:57.536412 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.537022 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.537026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.537028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.537127 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.537546 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.537549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.537551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.537792 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:57.537883 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.538515 (+ 632us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.538651 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:30:57.539986 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.524747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:46617 (local address 127.16.243.130:54633)
0618 22:30:57.525392 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.525481 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.527175 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.527779 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.527789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.528264 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:57.528935 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.528956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.530161 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.530165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.531275 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.531285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.531860 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.531864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.532059 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.532712 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:57.532733 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:57.534608 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:57.536542 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.536548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.536565 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.536916 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.537235 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.537238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.537240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.537433 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.537900 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:57.537908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:57.538228 (+ 320us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.538442 (+ 214us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.538837 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
I20260618 22:30:57.542600 3339 heartbeater.cc:347] Connected to a master server at 127.16.243.190:46617
I20260618 22:30:57.542855 3339 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:57.543464 3339 heartbeater.cc:511] Master 127.16.243.190:46617 requested a full tablet report, sending...
I20260618 22:30:57.544646 3014 ts_manager.cc:194] Registered new tserver with Master: d146d0a0012640d6aea8ff5ee2424be1 (127.16.243.130:35563)
I20260618 22:30:57.545404 3014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:54633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:30:57.557003 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:30:57.665242 3344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:57.665509 3344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:57.665570 3344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:57.669687 3344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:57.669755 3344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:57.669838 3344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:57.675194 3344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:46617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:57.676231 3344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:57.677240 3344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:57.686946 3350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:57.687081 3344 server_base.cc:1034] running on GCE node
W20260618 22:30:57.686950 3349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:57.686949 3352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:57.687583 3344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:57.688235 3344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:57.689378 3344 hybrid_clock.cc:648] HybridClock initialized: now 1781821857689341 us; error 55 us; skew 500 ppm
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:30:57.692108 3344 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:30:57.692994 3344 webserver.cc:460] Webserver started at http://127.16.243.131:41307/ using document root <none> and password file <none>
I20260618 22:30:57.693545 3344 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:57.693611 3344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:57.693811 3344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:57.695861 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "9eea7dcbc4f4422d954638f67d05f559"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "90ca9ef16086ea7d560b526e0bf2e05b"
server_key_iv: "a92ea76bbb4577991d2467914e31731c"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.696393 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "9eea7dcbc4f4422d954638f67d05f559"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "90ca9ef16086ea7d560b526e0bf2e05b"
server_key_iv: "a92ea76bbb4577991d2467914e31731c"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.700049 3344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:30:57.702503 3359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.703785 3344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:57.703908 3344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "9eea7dcbc4f4422d954638f67d05f559"
format_stamp: "Formatted at 2026-06-18 22:30:57 on dist-test-slave-jm5g"
server_key: "90ca9ef16086ea7d560b526e0bf2e05b"
server_key_iv: "a92ea76bbb4577991d2467914e31731c"
server_key_version: "encryptionkey@0"
I20260618 22:30:57.704013 3344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:57.724506 3344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:57.727650 3344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:57.727835 3344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:57.728417 3344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:57.729241 3344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:57.729288 3344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.729339 3344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:57.729368 3344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:57.737708 3344 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:41407
I20260618 22:30:57.737735 3472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:41407 every 8 connection(s)
I20260618 22:30:57.738631 3344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:57.743121 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3344
I20260618 22:30:57.743219 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:30:57.743474 17358 external_mini_cluster.cc:1428] Setting key bae0b4db4aacc0577c21784421d8ca71
Jun 18 22:30:57 dist-test-slave-jm5g krb5kdc[2969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821857, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:30:57.752436 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.740401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:55891 (local address 127.16.243.190:46617)
0618 22:30:57.740591 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.740594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.741423 (+ 829us) server_negotiation.cc:407] Connection header received
0618 22:30:57.742830 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.742833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.742913 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.742999 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:57.744356 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.745081 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.746320 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.746539 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.746643 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.749019 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:57.749022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:57.749024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:57.749049 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:57.750280 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.750852 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.750854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.750856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.750928 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.751224 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.751226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.751227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.751437 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:57.751518 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.752202 (+ 684us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.752311 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260618 22:30:57.753353 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.740784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:46617 (local address 127.16.243.131:55891)
0618 22:30:57.741240 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.741289 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.742623 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.743200 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.743210 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.743640 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:57.744214 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.744235 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.745216 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.745219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.746218 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.746227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.746678 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.746680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.746827 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.747368 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:30:57.747385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:57.748867 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:57.750394 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.750411 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.750428 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.750768 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.751005 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:30:57.751008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:30:57.751009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:30:57.751150 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:30:57.751642 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:57.751648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:57.751939 (+ 291us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.752151 (+ 212us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.752442 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260618 22:30:57.754504 3473 heartbeater.cc:347] Connected to a master server at 127.16.243.190:46617
I20260618 22:30:57.754741 3473 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:57.755194 3473 heartbeater.cc:511] Master 127.16.243.190:46617 requested a full tablet report, sending...
I20260618 22:30:57.756206 3014 ts_manager.cc:194] Registered new tserver with Master: 9eea7dcbc4f4422d954638f67d05f559 (127.16.243.131:41407)
I20260618 22:30:57.756778 3014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:55891
I20260618 22:30:57.757154 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:30:57.766419 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.759377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.16.243.190:46617)
0618 22:30:57.759504 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.759507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.759633 (+ 126us) server_negotiation.cc:407] Connection header received
0618 22:30:57.759747 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.759749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.759784 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.759840 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:57.760635 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.761162 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.761982 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.762209 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.762293 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.763707 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:57.763709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:57.763711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:30:57.763735 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:57.764857 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.765335 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.765337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.765339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.765415 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:30:57.765714 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:30:57.765717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:30:57.765718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:30:57.765924 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:57.766037 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.766221 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.766300 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:30:57.769299 3014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:30:57.772374 3014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:30:57.787447 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.780791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39413 (local address 127.0.0.1:47292)
0618 22:30:57.782005 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.782018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.782101 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.782590 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.782594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.782611 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:57.782701 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.782707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.784146 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.784150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.785778 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.785788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.786945 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.786949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.787049 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.787075 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.787127 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.787282 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"mutex_wait_us":378,"thread_start_us":105,"threads_started":1}
I20260618 22:30:57.787443 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.781608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.16.243.130:35563)
0618 22:30:57.782659 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.782672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.782695 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:30:57.782767 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.782772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.782930 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.783057 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:57.783276 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.784002 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.785938 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.786756 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.786872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.787018 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.787186 (+ 168us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.787283 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":138,"threads_started":1}
I20260618 22:30:57.787943 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.780791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:41407 (local address 127.0.0.1:56096)
0618 22:30:57.781619 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.781673 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.781800 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.782413 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.782419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.782441 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:57.782604 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.782612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.784087 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.784090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.786232 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.786241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.787349 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.787353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.787643 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.787667 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.787740 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.787805 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260618 22:30:57.788007 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.780925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.16.243.131:41407)
0618 22:30:57.781382 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.781394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.781942 (+ 548us) server_negotiation.cc:407] Connection header received
0618 22:30:57.782027 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.782033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.782254 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.782418 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:57.782742 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.783952 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.786346 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.787177 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.787288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.787433 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.787786 (+ 353us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.787884 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":121,"threads_started":1}
I20260618 22:30:57.788281 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.781003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:35563 (local address 127.0.0.1:49824)
0618 22:30:57.782289 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.782301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.782376 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.783061 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.783064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.783080 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:57.783181 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.783187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.784134 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.784137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.785802 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.785812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.786871 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.786874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.786996 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.787035 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.787143 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.787284 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":95,"thread_start_us":81,"threads_started":1}
I20260618 22:30:57.788789 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.781623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.16.243.129:39413)
0618 22:30:57.782158 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.782174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.782195 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:30:57.782259 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.782263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.782427 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.782847 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:57.782999 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.784041 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.785911 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.786793 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.787915 (+ 1122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.788038 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.788543 (+ 505us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.788655 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":71,"threads_started":1}
I20260618 22:30:57.790920 3273 tablet_service.cc:1467] Processing CreateTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 (DEFAULT_TABLE table=test-table [id=557b852c45a447578ca65fb1d55cef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:57.791009 3407 tablet_service.cc:1467] Processing CreateTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 (DEFAULT_TABLE table=test-table [id=557b852c45a447578ca65fb1d55cef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:57.791301 3139 tablet_service.cc:1467] Processing CreateTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 (DEFAULT_TABLE table=test-table [id=557b852c45a447578ca65fb1d55cef53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:57.791918 3273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a13cacf41542038850a4cfa4aab2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:57.792032 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a13cacf41542038850a4cfa4aab2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:57.792127 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a13cacf41542038850a4cfa4aab2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:57.798651 3490 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Bootstrap starting.
I20260618 22:30:57.800275 3492 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Bootstrap starting.
I20260618 22:30:57.800275 3491 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Bootstrap starting.
I20260618 22:30:57.800971 3490 tablet_bootstrap.cc:654] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:57.802028 3490 log.cc:824] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:57.802564 3491 tablet_bootstrap.cc:654] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:57.802564 3492 tablet_bootstrap.cc:654] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:57.803817 3492 log.cc:824] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:57.803817 3491 log.cc:824] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:57.803891 3490 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: No bootstrap required, opened a new log
I20260618 22:30:57.804193 3490 ts_tablet_manager.cc:1397] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:30:57.806455 3492 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: No bootstrap required, opened a new log
I20260618 22:30:57.806455 3491 tablet_bootstrap.cc:492] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: No bootstrap required, opened a new log
I20260618 22:30:57.806695 3491 ts_tablet_manager.cc:1397] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:30:57.806695 3492 ts_tablet_manager.cc:1397] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260618 22:30:57.807458 3490 raft_consensus.cc:348] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.807718 3490 raft_consensus.cc:374] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:57.807777 3490 raft_consensus.cc:729] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d146d0a0012640d6aea8ff5ee2424be1, State: Initialized, Role: FOLLOWER
I20260618 22:30:57.808194 3490 consensus_queue.cc:260] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.808880 3339 heartbeater.cc:503] Master 127.16.243.190:46617 was elected leader, sending a full tablet report...
I20260618 22:30:57.809145 3490 ts_tablet_manager.cc:1428] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260618 22:30:57.810377 3491 raft_consensus.cc:348] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.810627 3491 raft_consensus.cc:374] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:57.810675 3491 raft_consensus.cc:729] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7e01a78d841a8b9ba50fe37da21f8, State: Initialized, Role: FOLLOWER
I20260618 22:30:57.811038 3491 consensus_queue.cc:260] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.811133 3492 raft_consensus.cc:348] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.811422 3492 raft_consensus.cc:374] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:57.811480 3492 raft_consensus.cc:729] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eea7dcbc4f4422d954638f67d05f559, State: Initialized, Role: FOLLOWER
I20260618 22:30:57.811636 3205 heartbeater.cc:503] Master 127.16.243.190:46617 was elected leader, sending a full tablet report...
I20260618 22:30:57.811878 3492 consensus_queue.cc:260] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.811998 3491 ts_tablet_manager.cc:1428] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:30:57.812853 3206 tablet.cc:2369] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:30:57.813041 3473 heartbeater.cc:503] Master 127.16.243.190:46617 was elected leader, sending a full tablet report...
I20260618 22:30:57.813567 3492 ts_tablet_manager.cc:1428] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:30:57.894260 3496 raft_consensus.cc:482] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:57.894469 3496 raft_consensus.cc:504] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.895684 3496 leader_election.cc:288] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eea7dcbc4f4422d954638f67d05f559 (127.16.243.131:41407), 70d7e01a78d841a8b9ba50fe37da21f8 (127.16.243.129:39413)
I20260618 22:30:57.900496 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.896352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:59547 (local address 127.16.243.129:39413)
0618 22:30:57.896472 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.896476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.896489 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:30:57.896539 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.896541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.896581 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.896638 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:57.896908 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.897606 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.899265 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.899859 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.899987 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.900050 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.900295 (+ 245us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.900364 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:30:57.900496 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.895993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:47063 (local address 127.16.243.131:41407)
0618 22:30:57.896111 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.896114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.896129 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:30:57.896287 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.896290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.896350 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.896426 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:57.896769 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.897545 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.899265 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.899859 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.899987 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.900043 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.900299 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.900364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:30:57.900538 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.895913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:41407 (local address 127.16.243.130:47063)
0618 22:30:57.896062 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.896078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.896175 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.896551 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.896553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.896566 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:57.896672 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.896678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.897663 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.897667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.899153 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.899164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.900004 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.900009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.900149 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.900191 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.900270 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.900352 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":121}
I20260618 22:30:57.900547 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.895969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39413 (local address 127.16.243.130:59547)
0618 22:30:57.896285 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.896298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:57.896391 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:57.896667 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:57.896670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:57.896682 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:57.896795 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.896801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.897706 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.897709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.899154 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:57.899164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.900004 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.900009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:57.900152 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.900202 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:30:57.900273 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 22:30:57.900352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":85,"thread_start_us":139,"threads_started":1}
I20260618 22:30:57.901213 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a13cacf41542038850a4cfa4aab2b8" candidate_uuid: "d146d0a0012640d6aea8ff5ee2424be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eea7dcbc4f4422d954638f67d05f559" is_pre_election: true
I20260618 22:30:57.901480 3427 raft_consensus.cc:2454] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d146d0a0012640d6aea8ff5ee2424be1 in term 0.
I20260618 22:30:57.901939 3229 leader_election.cc:302] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eea7dcbc4f4422d954638f67d05f559, d146d0a0012640d6aea8ff5ee2424be1; no voters:
I20260618 22:30:57.901955 3159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a13cacf41542038850a4cfa4aab2b8" candidate_uuid: "d146d0a0012640d6aea8ff5ee2424be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" is_pre_election: true
I20260618 22:30:57.902236 3159 raft_consensus.cc:2454] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d146d0a0012640d6aea8ff5ee2424be1 in term 0.
I20260618 22:30:57.902249 3496 raft_consensus.cc:2781] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:30:57.902326 3496 raft_consensus.cc:482] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:57.902369 3496 raft_consensus.cc:3037] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:57.903501 3496 raft_consensus.cc:504] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.903836 3496 leader_election.cc:288] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [CANDIDATE]: Term 1 election: Requested vote from peers 9eea7dcbc4f4422d954638f67d05f559 (127.16.243.131:41407), 70d7e01a78d841a8b9ba50fe37da21f8 (127.16.243.129:39413)
I20260618 22:30:57.904165 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a13cacf41542038850a4cfa4aab2b8" candidate_uuid: "d146d0a0012640d6aea8ff5ee2424be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eea7dcbc4f4422d954638f67d05f559"
I20260618 22:30:57.904184 3159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a13cacf41542038850a4cfa4aab2b8" candidate_uuid: "d146d0a0012640d6aea8ff5ee2424be1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7e01a78d841a8b9ba50fe37da21f8"
I20260618 22:30:57.904281 3159 raft_consensus.cc:3037] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:57.904280 3427 raft_consensus.cc:3037] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:57.905437 3159 raft_consensus.cc:2454] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d146d0a0012640d6aea8ff5ee2424be1 in term 1.
I20260618 22:30:57.905475 3427 raft_consensus.cc:2454] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d146d0a0012640d6aea8ff5ee2424be1 in term 1.
I20260618 22:30:57.905778 3227 leader_election.cc:302] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 70d7e01a78d841a8b9ba50fe37da21f8, 9eea7dcbc4f4422d954638f67d05f559, d146d0a0012640d6aea8ff5ee2424be1; no voters:
I20260618 22:30:57.905954 3496 raft_consensus.cc:2781] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:57.906173 3496 raft_consensus.cc:686] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 1 LEADER]: Becoming Leader. State: Replica: d146d0a0012640d6aea8ff5ee2424be1, State: Running, Role: LEADER
I20260618 22:30:57.906471 3496 consensus_queue.cc:237] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } }
I20260618 22:30:57.909428 3012 catalog_manager.cc:5495] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d146d0a0012640d6aea8ff5ee2424be1 (127.16.243.130). New cstate: current_term: 1 leader_uuid: "d146d0a0012640d6aea8ff5ee2424be1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d146d0a0012640d6aea8ff5ee2424be1" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 35563 } health_report { overall_health: HEALTHY } } }
I20260618 22:30:57.943514 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.940157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.16.243.130:35563)
0618 22:30:57.940272 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:57.940275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:57.940288 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:30:57.940355 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:57.940357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:57.940411 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:57.940498 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:57.940687 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.941176 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.942059 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:57.942274 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:57.942357 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:57.942697 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:57.943099 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:30:57.943225 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:57.943365 (+ 140us) server_negotiation.cc:299] Negotiation successful
0618 22:30:57.943401 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:30:57.949724 3427 raft_consensus.cc:1261] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 1 FOLLOWER]: Refusing update from remote peer d146d0a0012640d6aea8ff5ee2424be1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:57.949723 3159 raft_consensus.cc:1261] T 41a13cacf41542038850a4cfa4aab2b8 P 70d7e01a78d841a8b9ba50fe37da21f8 [term 1 FOLLOWER]: Refusing update from remote peer d146d0a0012640d6aea8ff5ee2424be1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:57.950416 3500 consensus_queue.cc:1035] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d7e01a78d841a8b9ba50fe37da21f8" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:57.950621 3496 consensus_queue.cc:1035] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eea7dcbc4f4422d954638f67d05f559" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 41407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:57.958052 3505 mvcc.cc:205] Tried to move back new op lower bound from 7298342330155012096 to 7298342329987514368. Current Snapshot: MvccSnapshot[applied={T|T < 7298342330155012096}]
I20260618 22:30:57.967034 3014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:57.967238 3014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:57.969833 3014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 865b4ba15f074acdad5c4183dc331705: Sending DeleteTablet for 3 replicas of tablet 41a13cacf41542038850a4cfa4aab2b8
I20260618 22:30:57.970647 3139 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:47292
I20260618 22:30:57.970647 3273 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:49824
I20260618 22:30:57.970672 3407 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a13cacf41542038850a4cfa4aab2b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:57 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:56096
I20260618 22:30:57.971500 3513 tablet_replica.cc:330] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: stopping tablet replica
I20260618 22:30:57.971545 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3076
I20260618 22:30:57.971825 3513 raft_consensus.cc:2229] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:57.972213 3513 raft_consensus.cc:2258] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:57.973668 3513 ts_tablet_manager.cc:1905] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:57.974789 3514 tablet_replica.cc:330] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: stopping tablet replica
I20260618 22:30:57.975169 3514 raft_consensus.cc:2229] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:30:57.975668 3514 raft_consensus.cc:2258] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:57.976228 3513 ts_tablet_manager.cc:1918] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:57.976328 3513 log.cc:1196] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/41a13cacf41542038850a4cfa4aab2b8
I20260618 22:30:57.976650 3513 ts_tablet_manager.cc:1939] T 41a13cacf41542038850a4cfa4aab2b8 P 9eea7dcbc4f4422d954638f67d05f559: Deleting consensus metadata
I20260618 22:30:57.977252 3514 ts_tablet_manager.cc:1905] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:57.977854 3001 catalog_manager.cc:4841] TS 9eea7dcbc4f4422d954638f67d05f559 (127.16.243.131:41407): tablet 41a13cacf41542038850a4cfa4aab2b8 (table test-table [id=557b852c45a447578ca65fb1d55cef53]) successfully deleted
I20260618 22:30:57.979475 3514 ts_tablet_manager.cc:1918] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:57.979557 3514 log.cc:1196] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/41a13cacf41542038850a4cfa4aab2b8
I20260618 22:30:57.979811 3514 ts_tablet_manager.cc:1939] T 41a13cacf41542038850a4cfa4aab2b8 P d146d0a0012640d6aea8ff5ee2424be1: Deleting consensus metadata
I20260618 22:30:57.980787 2998 catalog_manager.cc:4841] TS d146d0a0012640d6aea8ff5ee2424be1 (127.16.243.130:35563): tablet 41a13cacf41542038850a4cfa4aab2b8 (table test-table [id=557b852c45a447578ca65fb1d55cef53]) successfully deleted
W20260618 22:30:57.983992 2999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:39413 (error 108)
I20260618 22:30:57.984164 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3210
I20260618 22:30:57.985904 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:57.985642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:39413 (local address 127.0.0.1:47294)
0618 22:30:57.985768 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:57.985828 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:30:57.986071 2999 catalog_manager.cc:4584] TS 70d7e01a78d841a8b9ba50fe37da21f8 (127.16.243.129:39413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41a13cacf41542038850a4cfa4aab2b8: Network error: Client connection negotiation failed: client connection to 127.16.243.129:39413: connect: Connection refused (error 111)
I20260618 22:30:57.989162 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3344
I20260618 22:30:57.993590 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 2982
2026-06-18T22:30:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2201 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-18T22:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:30:58Z Disabled control of system clock
I20260618 22:30:58.020601 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42847
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:42847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:30:58.127556 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:58.127823 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:58.127858 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:58.131866 3520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:30:58.131933 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:58.131948 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:58.131963 3520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:30:58.131978 3520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:30:58.136822 3520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:42847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:42847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:58.137908 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:58.139284 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:58.147394 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.147382 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.147418 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:58.147452 3520 server_base.cc:1034] running on GCE node
I20260618 22:30:58.148159 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:58.149199 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:58.150349 3520 hybrid_clock.cc:648] HybridClock initialized: now 1781821858150320 us; error 43 us; skew 500 ppm
I20260618 22:30:58.152105 3520 webserver.cc:460] Webserver started at http://127.16.243.190:40405/ using document root <none> and password file <none>
I20260618 22:30:58.152645 3520 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:58.152709 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:58.152887 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:58.154850 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "1d6b4600157444798c72b41bd18938f4"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "2213f58af65f2dd1bfe967757b8c74ee"
server_key_iv: "4277caa74a805a8a067ef250c09671c9"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.155385 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "1d6b4600157444798c72b41bd18938f4"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "2213f58af65f2dd1bfe967757b8c74ee"
server_key_iv: "4277caa74a805a8a067ef250c09671c9"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.158895 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:30:58.161264 3534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.162674 3520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:58.162792 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "1d6b4600157444798c72b41bd18938f4"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "2213f58af65f2dd1bfe967757b8c74ee"
server_key_iv: "4277caa74a805a8a067ef250c09671c9"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.162894 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:58.177441 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:58.178154 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:58.178326 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:58.184926 3520 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:42847
I20260618 22:30:58.185003 3586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:42847 every 8 connection(s)
I20260618 22:30:58.185886 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:30:58.189013 3587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:58.194993 3587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: Bootstrap starting.
I20260618 22:30:58.196002 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3520
I20260618 22:30:58.196107 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:30:58.196370 17358 external_mini_cluster.cc:1428] Setting key 0839dfa0dc7507fb95c34d5f51a65ec4
I20260618 22:30:58.197815 3587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:58.198627 3587 log.cc:824] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:58.200704 3587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: No bootstrap required, opened a new log
I20260618 22:30:58.204937 3587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } }
I20260618 22:30:58.205196 3587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:58.205260 3587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6b4600157444798c72b41bd18938f4, State: Initialized, Role: FOLLOWER
I20260618 22:30:58.205574 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.197557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.16.243.190:42847)
0618 22:30:58.198209 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.198223 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.198259 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:30:58.198966 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.198986 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.199327 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.199605 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:58.199976 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.200965 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.202172 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.202681 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.202804 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.203690 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:58.203695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:58.203708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:58.204094 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:58.204148 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:58.204167 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:58.204236 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.204423 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.204703 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260618 22:30:58.205721 3587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } }
I20260618 22:30:58.205868 3587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:30:58.205937 3587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:30:58.206038 3587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:58.207039 3587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } }
I20260618 22:30:58.207377 3587 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6b4600157444798c72b41bd18938f4; no voters:
I20260618 22:30:58.207688 3587 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:30:58.207830 3592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:58.208036 3592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1d6b4600157444798c72b41bd18938f4, State: Running, Role: LEADER
I20260618 22:30:58.208340 3592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } }
I20260618 22:30:58.209004 3587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:30:58.210394 3593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d6b4600157444798c72b41bd18938f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } } }
I20260618 22:30:58.210407 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6b4600157444798c72b41bd18938f4. Latest consensus state: current_term: 1 leader_uuid: "1d6b4600157444798c72b41bd18938f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6b4600157444798c72b41bd18938f4" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 42847 } } }
I20260618 22:30:58.210525 3593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:58.210541 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:30:58.211347 3601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:30:58.214713 3601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:30:58.220983 3601 catalog_manager.cc:1347] Generated new cluster ID: 32976fe578434c2abcea530e5a5e33b0
I20260618 22:30:58.221067 3601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:30:58.244921 3601 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:30:58.245937 3601 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:30:58.251856 3601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: Generated new TSK 0
I20260618 22:30:58.252550 3601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:30:58.262075 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--builtin_ntp_servers=127.16.243.148:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:30:58.371549 3611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:58.371801 3611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:58.371841 3611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:58.375840 3611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:58.375896 3611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:58.375955 3611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:30:58.380728 3611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:58.381800 3611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:58.383194 3611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:58.392324 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.392369 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.392324 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:58.392381 3611 server_base.cc:1034] running on GCE node
I20260618 22:30:58.392958 3611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:58.393651 3611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:58.394809 3611 hybrid_clock.cc:648] HybridClock initialized: now 1781821858394785 us; error 35 us; skew 500 ppm
I20260618 22:30:58.396556 3611 webserver.cc:460] Webserver started at http://127.16.243.129:46715/ using document root <none> and password file <none>
I20260618 22:30:58.397099 3611 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:58.397166 3611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:58.397363 3611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:58.399421 3611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "ab30224a23a84d269967dd550451e9cf"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "561a0e566dfc4a97695dfb12e0fe5f75"
server_key_iv: "3177b67fcf51c777f59c8f350fdc9466"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.399941 3611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "ab30224a23a84d269967dd550451e9cf"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "561a0e566dfc4a97695dfb12e0fe5f75"
server_key_iv: "3177b67fcf51c777f59c8f350fdc9466"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.403481 3611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:30:58.405753 3625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.406949 3611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:30:58.407050 3611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "ab30224a23a84d269967dd550451e9cf"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "561a0e566dfc4a97695dfb12e0fe5f75"
server_key_iv: "3177b67fcf51c777f59c8f350fdc9466"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.407141 3611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:58.430536 3611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:58.431255 3611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:58.431421 3611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:58.431988 3611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:58.432888 3611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:58.432942 3611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.432994 3611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:58.433022 3611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.440936 3611 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:41607
I20260618 22:30:58.440987 3738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:41607 every 8 connection(s)
I20260618 22:30:58.441789 3611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:30:58.448254 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3611
I20260618 22:30:58.448359 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:30:58.448601 17358 external_mini_cluster.cc:1428] Setting key 7c30247c47d660bd4377d138cad4755f
I20260618 22:30:58.450506 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--builtin_ntp_servers=127.16.243.148:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:30:58.451036 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.443608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:37199 (local address 127.16.243.190:42847)
0618 22:30:58.443751 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.443755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.444813 (+ 1058us) server_negotiation.cc:407] Connection header received
0618 22:30:58.446113 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.446115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.446161 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.446229 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:58.447164 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.447681 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.448942 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.449210 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.449321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.450454 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:58.450457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:58.450460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:58.450472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:58.450491 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:58.450499 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:58.450574 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.450840 (+ 266us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.450898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:30:58.452101 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.444016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42847 (local address 127.16.243.129:37199)
0618 22:30:58.444676 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.444718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.445931 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.446421 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.446429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.446516 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:58.447052 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.447064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.447807 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.447810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.448807 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.448817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.449361 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.449365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.449554 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.450188 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:58.450189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:58.450230 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:58.450304 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:58.450586 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:58.450702 (+ 116us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:58.450709 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.450829 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.451204 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260618 22:30:58.453271 3739 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42847
I20260618 22:30:58.453477 3739 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:58.453963 3739 heartbeater.cc:511] Master 127.16.243.190:42847 requested a full tablet report, sending...
I20260618 22:30:58.455521 3551 ts_manager.cc:194] Registered new tserver with Master: ab30224a23a84d269967dd550451e9cf (127.16.243.129:41607)
I20260618 22:30:58.457080 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.129:37199
W20260618 22:30:58.559883 3742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:58.560132 3742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:58.560168 3742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:58.564056 3742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:58.564105 3742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:58.564155 3742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:30:58.568930 3742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:58.569953 3742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:58.571365 3742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:58.580828 3747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.580835 3748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.580828 3750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:58.581027 3742 server_base.cc:1034] running on GCE node
I20260618 22:30:58.581593 3742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:58.582252 3742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:58.583432 3742 hybrid_clock.cc:648] HybridClock initialized: now 1781821858583392 us; error 53 us; skew 500 ppm
I20260618 22:30:58.585253 3742 webserver.cc:460] Webserver started at http://127.16.243.130:39461/ using document root <none> and password file <none>
I20260618 22:30:58.585837 3742 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:58.585906 3742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:58.586108 3742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:58.588172 3742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "adddfe115a584711b75a07b0f0b554dc"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "c0a53a36fe4c52a9059b9655f8894ecf"
server_key_iv: "5ca4603c9515f3b612767fd27964a52a"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.588747 3742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "adddfe115a584711b75a07b0f0b554dc"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "c0a53a36fe4c52a9059b9655f8894ecf"
server_key_iv: "5ca4603c9515f3b612767fd27964a52a"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.592471 3742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:30:58.595013 3756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.596252 3742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:58.596361 3742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "adddfe115a584711b75a07b0f0b554dc"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "c0a53a36fe4c52a9059b9655f8894ecf"
server_key_iv: "5ca4603c9515f3b612767fd27964a52a"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.596459 3742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:58.622339 3742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:58.623131 3742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:58.623302 3742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:58.623888 3742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:58.624823 3742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:58.624868 3742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.624919 3742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:58.624945 3742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.633200 3742 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:33139
I20260618 22:30:58.633241 3869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:33139 every 8 connection(s)
I20260618 22:30:58.634099 3742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:30:58.636715 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3742
I20260618 22:30:58.636829 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:30:58.637094 17358 external_mini_cluster.cc:1428] Setting key ea8f101cd46678832fb1bc7fd2a364e5
I20260618 22:30:58.639051 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--builtin_ntp_servers=127.16.243.148:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:30:58.643755 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.635914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:49327 (local address 127.16.243.190:42847)
0618 22:30:58.636072 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.636075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.637242 (+ 1167us) server_negotiation.cc:407] Connection header received
0618 22:30:58.638771 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.638774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.638825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.638894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:58.639819 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.640556 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.641703 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.642017 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.642123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.643263 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:58.643268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:58.643270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:58.643286 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:58.643309 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:58.643320 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:58.643399 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.643525 (+ 126us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.643599 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:30:58.644550 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.636342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42847 (local address 127.16.243.130:49327)
0618 22:30:58.637077 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.637131 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.638542 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.639063 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.639073 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.639166 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:58.639688 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.639701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.640687 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.640690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.641586 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.641596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.642149 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.642151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.642326 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.642963 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:58.642964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:58.643005 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:58.643098 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:58.643398 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:58.643404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:58.643408 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.643503 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.643740 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260618 22:30:58.645715 3870 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42847
I20260618 22:30:58.645932 3870 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:58.646426 3870 heartbeater.cc:511] Master 127.16.243.190:42847 requested a full tablet report, sending...
I20260618 22:30:58.647454 3551 ts_manager.cc:194] Registered new tserver with Master: adddfe115a584711b75a07b0f0b554dc (127.16.243.130:33139)
I20260618 22:30:58.648006 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.130:49327
W20260618 22:30:58.745572 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:30:58.745814 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:30:58.745853 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:30:58.749761 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:30:58.749810 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:30:58.749873 3873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:30:58.754662 3873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:30:58.755663 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:30:58.756971 3873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:30:58.765993 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:58.766052 3873 server_base.cc:1034] running on GCE node
W20260618 22:30:58.766013 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:30:58.766000 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:30:58.766654 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:30:58.767243 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:30:58.768385 3873 hybrid_clock.cc:648] HybridClock initialized: now 1781821858768359 us; error 38 us; skew 500 ppm
I20260618 22:30:58.770081 3873 webserver.cc:460] Webserver started at http://127.16.243.131:34143/ using document root <none> and password file <none>
I20260618 22:30:58.770632 3873 fs_manager.cc:359] Metadata directory not provided
I20260618 22:30:58.770694 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:30:58.770879 3873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:30:58.772851 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "096202effc2dbe6be95314cf431583c7"
server_key_iv: "aa51ec90d917aa2869e845a77353e124"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.773353 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "096202effc2dbe6be95314cf431583c7"
server_key_iv: "aa51ec90d917aa2869e845a77353e124"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.776712 3873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260618 22:30:58.779035 3887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.780150 3873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:30:58.780247 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b"
format_stamp: "Formatted at 2026-06-18 22:30:58 on dist-test-slave-jm5g"
server_key: "096202effc2dbe6be95314cf431583c7"
server_key_iv: "aa51ec90d917aa2869e845a77353e124"
server_key_version: "encryptionkey@0"
I20260618 22:30:58.780332 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:30:58.799630 3873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:30:58.800292 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:30:58.800453 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:30:58.801025 3873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:30:58.801929 3873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:30:58.801975 3873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.802022 3873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:30:58.802050 3873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:30:58.810066 3873 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:35699
I20260618 22:30:58.810130 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:35699 every 8 connection(s)
I20260618 22:30:58.811012 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:30:58.814975 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 3873
I20260618 22:30:58.815101 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:30:58.815384 17358 external_mini_cluster.cc:1428] Setting key 234828c5d6079441c3793ee5693fa9ed
I20260618 22:30:58.820112 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.812819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:53981 (local address 127.16.243.190:42847)
0618 22:30:58.812975 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.812979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.813988 (+ 1009us) server_negotiation.cc:407] Connection header received
0618 22:30:58.815581 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.815584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.815631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.815714 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:58.816788 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.817460 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.818521 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.818749 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.818814 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.819756 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:58.819759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:58.819761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:58.819771 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:58.819788 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:58.819795 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:58.819847 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.819954 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.820001 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:30:58.820961 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.813222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:42847 (local address 127.16.243.131:53981)
0618 22:30:58.813847 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.813887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.815348 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.815875 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.815884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.815993 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:30:58.816661 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.816680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.817580 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.817583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.818433 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.818442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.818859 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.818862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.818997 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.819519 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:30:58.819521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:30:58.819573 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:30:58.819634 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:30:58.819855 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:30:58.819860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:30:58.819864 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.819952 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.820167 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260618 22:30:58.822005 4001 heartbeater.cc:347] Connected to a master server at 127.16.243.190:42847
I20260618 22:30:58.822197 4001 heartbeater.cc:464] Registering TS with master...
I20260618 22:30:58.822702 4001 heartbeater.cc:511] Master 127.16.243.190:42847 requested a full tablet report, sending...
I20260618 22:30:58.823580 3551 ts_manager.cc:194] Registered new tserver with Master: 1b31cbdf16fa47949c2bf5da57e2ff0b (127.16.243.131:35699)
I20260618 22:30:58.824105 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.131:53981
I20260618 22:30:58.828797 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:30:58.833829 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.830586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.16.243.190:42847)
0618 22:30:58.830697 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.830701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.830859 (+ 158us) server_negotiation.cc:407] Connection header received
0618 22:30:58.830926 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.830928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.830968 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.831039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:30:58.831239 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.831734 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.832454 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.832655 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.832719 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.833419 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:30:58.833421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:30:58.833423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:30:58.833432 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:30:58.833445 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:30:58.833451 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:30:58.833526 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.833696 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.833729 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:30:58.836563 3551 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:30:58.839437 3551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:30:58.853214 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.847549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.16.243.130:33139)
0618 22:30:58.848146 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.848156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.848529 (+ 373us) server_negotiation.cc:407] Connection header received
0618 22:30:58.848812 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.848818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.849030 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.849153 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:58.849690 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850387 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.851891 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.852590 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.852716 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.852896 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.852957 (+ 61us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.853054 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":105,"threads_started":1}
I20260618 22:30:58.853205 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.847327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:33139 (local address 127.0.0.1:37058)
0618 22:30:58.848386 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.848440 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.848645 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.849320 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.849322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.849347 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:58.849551 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.849558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850510 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.850513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.851774 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.851783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.852726 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.852729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.852810 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.852841 (+ 31us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.852928 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.853047 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":115,"spinlock_wait_cycles":8960,"thread_start_us":92,"threads_started":1}
I20260618 22:30:58.854019 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.847389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.16.243.131:35699)
0618 22:30:58.847960 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.847973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.848529 (+ 556us) server_negotiation.cc:407] Connection header received
0618 22:30:58.848812 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.848818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.849029 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.849153 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:58.849670 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850778 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.852146 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.852881 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.853011 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.853160 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.853755 (+ 595us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.853857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":131,"threads_started":1}
I20260618 22:30:58.854316 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.847250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35699 (local address 127.0.0.1:50822)
0618 22:30:58.848386 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.848440 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.848645 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.849320 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.849322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.849349 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:58.849560 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.849565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850889 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.850892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.852031 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.852039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.853025 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.853028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.853126 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.853148 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.853216 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.853276 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":93,"spinlock_wait_cycles":9728,"thread_start_us":89,"threads_started":1}
I20260618 22:30:58.854867 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.848036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:41607 (local address 127.0.0.1:58682)
0618 22:30:58.848574 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.848588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.848675 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.849153 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.849158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.849179 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:58.849302 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.849309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850676 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.850679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.852028 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.852039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.854464 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.854468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.854586 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.854609 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.854677 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.854745 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260618 22:30:58.855294 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.848195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.16.243.129:41607)
0618 22:30:58.848772 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.848785 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.848804 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:30:58.848862 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.848865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.849018 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.849149 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:58.849409 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.850511 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.853222 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.853848 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.853974 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.854135 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.855084 (+ 949us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.855174 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":109,"threads_started":1}
I20260618 22:30:58.856495 3804 tablet_service.cc:1467] Processing CreateTablet for tablet ac6dd638294742b085079cd1a8f0c575 (DEFAULT_TABLE table=test-table [id=4a06196819834181be93d4e44b1bb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:58.856817 3935 tablet_service.cc:1467] Processing CreateTablet for tablet ac6dd638294742b085079cd1a8f0c575 (DEFAULT_TABLE table=test-table [id=4a06196819834181be93d4e44b1bb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:58.857635 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6dd638294742b085079cd1a8f0c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:58.857652 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6dd638294742b085079cd1a8f0c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:58.859373 3673 tablet_service.cc:1467] Processing CreateTablet for tablet ac6dd638294742b085079cd1a8f0c575 (DEFAULT_TABLE table=test-table [id=4a06196819834181be93d4e44b1bb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:30:58.860154 3673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac6dd638294742b085079cd1a8f0c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:30:58.864729 4019 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Bootstrap starting.
I20260618 22:30:58.865131 4018 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Bootstrap starting.
I20260618 22:30:58.865859 4020 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Bootstrap starting.
I20260618 22:30:58.867357 4018 tablet_bootstrap.cc:654] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:58.867699 4020 tablet_bootstrap.cc:654] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:58.868196 4018 log.cc:824] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:58.868321 4020 log.cc:824] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:58.868494 4019 tablet_bootstrap.cc:654] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Neither blocks nor log segments found. Creating new log.
I20260618 22:30:58.869256 4019 log.cc:824] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:30:58.870348 4018 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: No bootstrap required, opened a new log
I20260618 22:30:58.870447 4020 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: No bootstrap required, opened a new log
I20260618 22:30:58.870539 4018 ts_tablet_manager.cc:1397] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:30:58.870659 4020 ts_tablet_manager.cc:1397] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:30:58.871079 4019 tablet_bootstrap.cc:492] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: No bootstrap required, opened a new log
I20260618 22:30:58.871243 4019 ts_tablet_manager.cc:1397] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260618 22:30:58.873610 4018 raft_consensus.cc:348] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.873813 4018 raft_consensus.cc:374] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:58.873862 4018 raft_consensus.cc:729] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b31cbdf16fa47949c2bf5da57e2ff0b, State: Initialized, Role: FOLLOWER
I20260618 22:30:58.874257 4018 consensus_queue.cc:260] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.874697 4020 raft_consensus.cc:348] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.874891 4001 heartbeater.cc:503] Master 127.16.243.190:42847 was elected leader, sending a full tablet report...
I20260618 22:30:58.874944 4020 raft_consensus.cc:374] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:58.875000 4020 raft_consensus.cc:729] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab30224a23a84d269967dd550451e9cf, State: Initialized, Role: FOLLOWER
I20260618 22:30:58.875172 4018 ts_tablet_manager.cc:1428] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260618 22:30:58.875396 4020 consensus_queue.cc:260] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.876278 4020 ts_tablet_manager.cc:1428] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:30:58.876216 4019 raft_consensus.cc:348] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.876447 4019 raft_consensus.cc:374] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:30:58.876500 4019 raft_consensus.cc:729] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adddfe115a584711b75a07b0f0b554dc, State: Initialized, Role: FOLLOWER
I20260618 22:30:58.876602 3739 heartbeater.cc:503] Master 127.16.243.190:42847 was elected leader, sending a full tablet report...
I20260618 22:30:58.876879 4019 consensus_queue.cc:260] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.877422 3870 heartbeater.cc:503] Master 127.16.243.190:42847 was elected leader, sending a full tablet report...
I20260618 22:30:58.877671 4019 ts_tablet_manager.cc:1428] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260618 22:30:58.885035 3871 tablet.cc:2369] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:30:58.942979 3740 tablet.cc:2369] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:30:58.964838 4025 raft_consensus.cc:482] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:30:58.965046 4025 raft_consensus.cc:504] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.966130 4025 leader_election.cc:288] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b31cbdf16fa47949c2bf5da57e2ff0b (127.16.243.131:35699), adddfe115a584711b75a07b0f0b554dc (127.16.243.130:33139)
I20260618 22:30:58.970688 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.966367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:35699 (local address 127.16.243.129:47269)
0618 22:30:58.966660 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.966674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.966793 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.967173 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.967176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.967195 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:58.967307 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.967313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.968189 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.968192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.969631 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.969640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.970254 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.970258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.970350 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.970372 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.970434 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.970508 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":80,"thread_start_us":107,"threads_started":1}
I20260618 22:30:58.970707 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.966501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:47269 (local address 127.16.243.131:35699)
0618 22:30:58.966648 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.966652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.966755 (+ 103us) server_negotiation.cc:407] Connection header received
0618 22:30:58.966930 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.966934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.966983 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.967032 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:58.967409 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.968098 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.969759 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.970159 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.970254 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.970292 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.970490 (+ 198us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.970589 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260618 22:30:58.971238 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6dd638294742b085079cd1a8f0c575" candidate_uuid: "ab30224a23a84d269967dd550451e9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" is_pre_election: true
I20260618 22:30:58.971433 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.966367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:33139 (local address 127.16.243.129:43697)
0618 22:30:58.966963 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:58.966975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:30:58.967115 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:30:58.967376 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:30:58.967379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:30:58.967396 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:30:58.967515 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.967521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.968299 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.968302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.969629 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:30:58.969640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.971062 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.971066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:30:58.971143 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.971160 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:30:58.971223 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 22:30:58.971287 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":367,"thread_start_us":63,"threads_started":1}
I20260618 22:30:58.971422 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:58.966495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:43697 (local address 127.16.243.130:33139)
0618 22:30:58.966648 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:58.966653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:58.967068 (+ 415us) server_negotiation.cc:407] Connection header received
0618 22:30:58.967219 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:58.967223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:58.967284 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:58.967365 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:30:58.967630 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.968207 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.970340 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:58.970835 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:58.970963 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:58.971010 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:58.971253 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:30:58.971323 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260618 22:30:58.971562 3955 raft_consensus.cc:2454] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab30224a23a84d269967dd550451e9cf in term 0.
I20260618 22:30:58.972033 3626 leader_election.cc:302] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b31cbdf16fa47949c2bf5da57e2ff0b, ab30224a23a84d269967dd550451e9cf; no voters:
I20260618 22:30:58.972062 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6dd638294742b085079cd1a8f0c575" candidate_uuid: "ab30224a23a84d269967dd550451e9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adddfe115a584711b75a07b0f0b554dc" is_pre_election: true
I20260618 22:30:58.972282 4025 raft_consensus.cc:2781] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:30:58.972306 3824 raft_consensus.cc:2454] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab30224a23a84d269967dd550451e9cf in term 0.
I20260618 22:30:58.972365 4025 raft_consensus.cc:482] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:30:58.972440 4025 raft_consensus.cc:3037] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:58.973455 4025 raft_consensus.cc:504] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.973783 4025 leader_election.cc:288] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [CANDIDATE]: Term 1 election: Requested vote from peers 1b31cbdf16fa47949c2bf5da57e2ff0b (127.16.243.131:35699), adddfe115a584711b75a07b0f0b554dc (127.16.243.130:33139)
I20260618 22:30:58.974087 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6dd638294742b085079cd1a8f0c575" candidate_uuid: "ab30224a23a84d269967dd550451e9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b"
I20260618 22:30:58.974148 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac6dd638294742b085079cd1a8f0c575" candidate_uuid: "ab30224a23a84d269967dd550451e9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adddfe115a584711b75a07b0f0b554dc"
I20260618 22:30:58.974198 3955 raft_consensus.cc:3037] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:58.974272 3824 raft_consensus.cc:3037] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:30:58.975229 3955 raft_consensus.cc:2454] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab30224a23a84d269967dd550451e9cf in term 1.
I20260618 22:30:58.975286 3824 raft_consensus.cc:2454] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab30224a23a84d269967dd550451e9cf in term 1.
I20260618 22:30:58.975502 3626 leader_election.cc:302] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b31cbdf16fa47949c2bf5da57e2ff0b, ab30224a23a84d269967dd550451e9cf; no voters:
I20260618 22:30:58.975749 4025 raft_consensus.cc:2781] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:30:58.975924 4025 raft_consensus.cc:686] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [term 1 LEADER]: Becoming Leader. State: Replica: ab30224a23a84d269967dd550451e9cf, State: Running, Role: LEADER
I20260618 22:30:58.976219 4025 consensus_queue.cc:237] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } }
I20260618 22:30:58.979214 3551 catalog_manager.cc:5495] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ab30224a23a84d269967dd550451e9cf (127.16.243.129). New cstate: current_term: 1 leader_uuid: "ab30224a23a84d269967dd550451e9cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab30224a23a84d269967dd550451e9cf" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 41607 } health_report { overall_health: HEALTHY } } }
I20260618 22:30:59.005596 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:59.002162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.16.243.129:41607)
0618 22:30:59.002284 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:30:59.002289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:30:59.002304 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:30:59.002420 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:30:59.002423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:30:59.002484 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:30:59.002604 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:30:59.002834 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:59.003359 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:59.004199 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:30:59.004416 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:30:59.004502 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:30:59.004793 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:30:59.005202 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:30:59.005328 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:30:59.005429 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 22:30:59.005462 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:30:59.011349 3824 raft_consensus.cc:1261] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 1 FOLLOWER]: Refusing update from remote peer ab30224a23a84d269967dd550451e9cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:59.011350 3955 raft_consensus.cc:1261] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 1 FOLLOWER]: Refusing update from remote peer ab30224a23a84d269967dd550451e9cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:30:59.012045 4029 consensus_queue.cc:1035] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "adddfe115a584711b75a07b0f0b554dc" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:59.012233 4025 consensus_queue.cc:1035] T ac6dd638294742b085079cd1a8f0c575 P ab30224a23a84d269967dd550451e9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31cbdf16fa47949c2bf5da57e2ff0b" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 35699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:30:59.028378 3551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:59.028529 3551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:30:59.030889 3551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d6b4600157444798c72b41bd18938f4: Sending DeleteTablet for 3 replicas of tablet ac6dd638294742b085079cd1a8f0c575
I20260618 22:30:59.031621 3673 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6dd638294742b085079cd1a8f0c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:59 UTC) from {username='slave'} at 127.0.0.1:58682
I20260618 22:30:59.031625 3935 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6dd638294742b085079cd1a8f0c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:59 UTC) from {username='slave'} at 127.0.0.1:50822
I20260618 22:30:59.032110 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet ac6dd638294742b085079cd1a8f0c575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:30:59 UTC) from {username='slave'} at 127.0.0.1:37058
I20260618 22:30:59.032284 4042 tablet_replica.cc:330] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: stopping tablet replica
I20260618 22:30:59.032379 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3611
I20260618 22:30:59.032963 4044 tablet_replica.cc:330] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: stopping tablet replica
I20260618 22:30:59.033252 4044 raft_consensus.cc:2229] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:59.033262 4042 raft_consensus.cc:2229] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:30:59.033545 4044 raft_consensus.cc:2258] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:59.033546 4042 raft_consensus.cc:2258] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:30:59.035063 4042 ts_tablet_manager.cc:1905] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:59.035063 4044 ts_tablet_manager.cc:1905] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:30:59.037235 4042 ts_tablet_manager.cc:1918] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:59.037315 4042 log.cc:1196] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/ac6dd638294742b085079cd1a8f0c575
I20260618 22:30:59.037565 4042 ts_tablet_manager.cc:1939] T ac6dd638294742b085079cd1a8f0c575 P 1b31cbdf16fa47949c2bf5da57e2ff0b: Deleting consensus metadata
I20260618 22:30:59.037616 4044 ts_tablet_manager.cc:1918] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:30:59.037703 4044 log.cc:1196] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/ac6dd638294742b085079cd1a8f0c575
I20260618 22:30:59.037972 4044 ts_tablet_manager.cc:1939] T ac6dd638294742b085079cd1a8f0c575 P adddfe115a584711b75a07b0f0b554dc: Deleting consensus metadata
I20260618 22:30:59.038709 3535 catalog_manager.cc:4841] TS 1b31cbdf16fa47949c2bf5da57e2ff0b (127.16.243.131:35699): tablet ac6dd638294742b085079cd1a8f0c575 (table test-table [id=4a06196819834181be93d4e44b1bb67f]) successfully deleted
I20260618 22:30:59.039023 3538 catalog_manager.cc:4841] TS adddfe115a584711b75a07b0f0b554dc (127.16.243.130:33139): tablet ac6dd638294742b085079cd1a8f0c575 (table test-table [id=4a06196819834181be93d4e44b1bb67f]) successfully deleted
W20260618 22:30:59.039414 3537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:41607 (error 108)
I20260618 22:30:59.039628 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3742
I20260618 22:30:59.041895 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:30:59.041614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:41607 (local address 127.0.0.1:58704)
0618 22:30:59.041746 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 22:30:59.041815 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260618 22:30:59.042063 3537 catalog_manager.cc:4584] TS ab30224a23a84d269967dd550451e9cf (127.16.243.129:41607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac6dd638294742b085079cd1a8f0c575: Network error: Client connection negotiation failed: client connection to 127.16.243.129:41607: connect: Connection refused (error 111)
I20260618 22:30:59.045033 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3873
I20260618 22:30:59.049124 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 3520
2026-06-18T22:30:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:30:59 dist-test-slave-jm5g krb5kdc[4048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:30:59 dist-test-slave-jm5g krb5kdc[4048](info): set up 2 sockets
Jun 18 22:30:59 dist-test-slave-jm5g krb5kdc[4048](info): commencing operation
krb5kdc: starting...
W20260618 22:31:00.010299 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:00.123222 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37547
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:37547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:00.232767 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:00.233055 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:00.233096 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:00.237162 4061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:00.237229 4061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:00.237247 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:00.237263 4061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:00.237277 4061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:00.242684 4061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:37547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:00.243810 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:00.244851 4061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:00.253343 4061 server_base.cc:1034] running on GCE node
W20260618 22:31:00.253293 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.253304 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.253302 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:00.254163 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:00.255349 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:00.256537 4061 hybrid_clock.cc:648] HybridClock initialized: now 1781821860256520 us; error 34 us; skew 500 ppm
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:00.259207 4061 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:00.260102 4061 webserver.cc:460] Webserver started at http://127.16.243.190:33481/ using document root <none> and password file <none>
I20260618 22:31:00.260635 4061 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:00.260699 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:00.260903 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:00.262975 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "76f7f1194ddf4102922be6ce1d995fe5"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "2157a2ff011e064796fcc2c0d011ac0f"
server_key_iv: "2f482e0134a88f15090d07430782a440"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.263520 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "76f7f1194ddf4102922be6ce1d995fe5"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "2157a2ff011e064796fcc2c0d011ac0f"
server_key_iv: "2f482e0134a88f15090d07430782a440"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.267164 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:00.269575 4076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.270872 4061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:00.270991 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "76f7f1194ddf4102922be6ce1d995fe5"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "2157a2ff011e064796fcc2c0d011ac0f"
server_key_iv: "2f482e0134a88f15090d07430782a440"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.271100 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:00.293829 4061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:00.296638 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:00.296819 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:00.303726 4061 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:37547
I20260618 22:31:00.303745 4128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:37547 every 8 connection(s)
I20260618 22:31:00.304678 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:00.307430 4129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:00.309265 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4061
I20260618 22:31:00.309386 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:00.309624 17358 external_mini_cluster.cc:1428] Setting key 0b7d88d52b342c6dbcd6e8eafa3b8625
I20260618 22:31:00.313640 4129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: Bootstrap starting.
I20260618 22:31:00.315897 4129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:00.316619 4129 log.cc:824] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:00.318858 4129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: No bootstrap required, opened a new log
I20260618 22:31:00.323498 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:00.310973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.16.243.190:37547)
0618 22:31:00.311386 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:00.311401 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:00.311436 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:00.312078 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:00.312095 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:00.312416 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:00.312747 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:00.313500 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.314486 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.315329 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.315795 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.315933 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:00.318397 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:00.318404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:00.318421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:00.318454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:00.320395 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.320841 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.320847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.320853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.321003 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.321257 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.321260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.321261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.321775 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:00.321950 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:00.322177 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:31:00.322481 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260618 22:31:00.323382 4129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } }
I20260618 22:31:00.323662 4129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:00.323743 4129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f7f1194ddf4102922be6ce1d995fe5, State: Initialized, Role: FOLLOWER
I20260618 22:31:00.324156 4129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } }
I20260618 22:31:00.324280 4129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:00.324342 4129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:00.324434 4129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:00.325557 4129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } }
I20260618 22:31:00.325906 4129 leader_election.cc:302] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f7f1194ddf4102922be6ce1d995fe5; no voters:
I20260618 22:31:00.326217 4129 leader_election.cc:288] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:00.326311 4134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:00.326503 4134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 76f7f1194ddf4102922be6ce1d995fe5, State: Running, Role: LEADER
I20260618 22:31:00.326843 4134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } }
I20260618 22:31:00.327333 4129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:00.328470 4136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f7f1194ddf4102922be6ce1d995fe5. Latest consensus state: current_term: 1 leader_uuid: "76f7f1194ddf4102922be6ce1d995fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } } }
I20260618 22:31:00.328564 4135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f7f1194ddf4102922be6ce1d995fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f7f1194ddf4102922be6ce1d995fe5" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37547 } } }
I20260618 22:31:00.328696 4135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:00.328583 4136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5 [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:00.332048 4149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:00.332120 4149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:00.332192 4144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:00.332857 4144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:00.338748 4144 catalog_manager.cc:1347] Generated new cluster ID: 5bd7c095efab46d5b73be57c918f15d6
I20260618 22:31:00.338830 4144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:00.355289 4144 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:00.356278 4144 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:00.369112 4144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: Generated new TSK 0
I20260618 22:31:00.370013 4144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
Entry for principal kudu/127.16.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab.
I20260618 22:31:00.399559 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:00.509310 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:00.509629 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:00.509681 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:00.513823 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:00.513896 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:00.513962 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:31:00.519381 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.129.keytab
--principal=kudu/127.16.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:00.520428 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:00.521433 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:00.532266 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.532258 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.532289 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:00.532749 4155 server_base.cc:1034] running on GCE node
I20260618 22:31:00.533115 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:00.533702 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:00.534861 4155 hybrid_clock.cc:648] HybridClock initialized: now 1781821860534833 us; error 37 us; skew 500 ppm
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:00.537448 4155 init.cc:377] Logged in from keytab as kudu/127.16.243.129@KRBTEST.COM (short username kudu)
I20260618 22:31:00.538322 4155 webserver.cc:460] Webserver started at http://127.16.243.129:46081/ using document root <none> and password file <none>
I20260618 22:31:00.538899 4155 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:00.538965 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:00.539173 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:00.541172 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "1e8946884e9342e4a312c8b1714f3081"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "11ee484e4dc61bfc4fafdb2e99da15b1"
server_key_iv: "92b8e723b1276b875e6085129981fee3"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.541687 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "1e8946884e9342e4a312c8b1714f3081"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "11ee484e4dc61bfc4fafdb2e99da15b1"
server_key_iv: "92b8e723b1276b875e6085129981fee3"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.545310 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:31:00.547778 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.549149 4155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:31:00.549255 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "1e8946884e9342e4a312c8b1714f3081"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "11ee484e4dc61bfc4fafdb2e99da15b1"
server_key_iv: "92b8e723b1276b875e6085129981fee3"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.549350 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:00.577739 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:00.580955 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:00.581146 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:00.581753 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:00.582630 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:00.582680 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.582734 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:00.582758 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.591245 4155 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:44159
I20260618 22:31:00.591279 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:44159 every 8 connection(s)
I20260618 22:31:00.592144 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:31:00.595760 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4155
I20260618 22:31:00.595888 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:31:00.596161 17358 external_mini_cluster.cc:1428] Setting key 3bc4626467ec31d66585f104b3f03f9b
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.129@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.243.130@KRBTEST.COM; defaulting to no policy
I20260618 22:31:00.606783 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:00.593963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:59183 (local address 127.16.243.190:37547)
0618 22:31:00.594111 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:00.594115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:00.594963 (+ 848us) server_negotiation.cc:407] Connection header received
0618 22:31:00.596382 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:00.596386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:00.596446 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:00.596531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:00.598029 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.598801 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.600077 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.600388 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.600489 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:00.603197 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:00.603201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:00.603204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:00.603235 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:00.604776 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.605279 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.605283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.605285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.605367 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.605672 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.605676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.605678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.605902 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:00.605980 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:00.606496 (+ 516us) server_negotiation.cc:299] Negotiation successful
0618 22:31:00.606656 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:31:00.607667 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:00.594338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37547 (local address 127.16.243.129:59183)
0618 22:31:00.594808 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:00.594853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:00.596131 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:00.596880 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:00.596888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:00.597251 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:00.597871 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:00.597887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.598959 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.598962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:00.599943 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:00.599952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.600555 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.600559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:00.600735 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:00.601321 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:00.601338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:00.603009 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:00.604905 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:00.604910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:00.604924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:00.605177 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:00.605454 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:00.605456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:00.605457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:00.605594 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:00.605999 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:00.606004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:00.606256 (+ 252us) client_negotiation.cc:769] Sending connection context
0618 22:31:00.606417 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:31:00.606780 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.130@KRBTEST.COM" created.
I20260618 22:31:00.608927 4284 heartbeater.cc:347] Connected to a master server at 127.16.243.190:37547
I20260618 22:31:00.609169 4284 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:00.609699 4284 heartbeater.cc:511] Master 127.16.243.190:37547 requested a full tablet report, sending...
I20260618 22:31:00.611239 4093 ts_manager.cc:194] Registered new tserver with Master: 1e8946884e9342e4a312c8b1714f3081 (127.16.243.129:44159)
I20260618 22:31:00.612362 4093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.129@KRBTEST.COM'} at 127.16.243.129:59183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
Entry for principal kudu/127.16.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab.
I20260618 22:31:00.621315 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:00.730841 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:00.731105 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:00.731168 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:00.735265 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:00.735334 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:00.735421 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:31:00.740886 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.130.keytab
--principal=kudu/127.16.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:00.741971 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:00.743063 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:00.753181 4289 server_base.cc:1034] running on GCE node
W20260618 22:31:00.753166 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.753391 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.753186 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:00.753893 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:00.754539 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:00.755753 4289 hybrid_clock.cc:648] HybridClock initialized: now 1781821860755729 us; error 40 us; skew 500 ppm
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:00.758672 4289 init.cc:377] Logged in from keytab as kudu/127.16.243.130@KRBTEST.COM (short username kudu)
I20260618 22:31:00.759538 4289 webserver.cc:460] Webserver started at http://127.16.243.130:43509/ using document root <none> and password file <none>
I20260618 22:31:00.760085 4289 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:00.760149 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:00.760342 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:00.762349 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "bef9527e3277446188d02fdafccaf701"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "678a9a34a16ee58f1c2c96b648c817a9"
server_key_iv: "c4b58a7b75773cca6ed9447a0903da88"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.762917 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "bef9527e3277446188d02fdafccaf701"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "678a9a34a16ee58f1c2c96b648c817a9"
server_key_iv: "c4b58a7b75773cca6ed9447a0903da88"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.766636 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:00.768985 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.770212 4289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:00.770334 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "bef9527e3277446188d02fdafccaf701"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "678a9a34a16ee58f1c2c96b648c817a9"
server_key_iv: "c4b58a7b75773cca6ed9447a0903da88"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.770437 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:00.795648 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:00.798861 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:00.799058 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:00.799659 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:00.800436 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:00.800477 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.800515 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:00.800535 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.808698 4289 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:39333
I20260618 22:31:00.808789 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:39333 every 8 connection(s)
I20260618 22:31:00.809592 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:31:00.817777 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4289
I20260618 22:31:00.817914 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:31:00.818213 17358 external_mini_cluster.cc:1428] Setting key 4da0b01e8b44cfa53606bc9c62e23d83
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.130@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:00.823902 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:00.811466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:39593 (local address 127.16.243.190:37547)
0618 22:31:00.811637 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:00.811641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:00.812439 (+ 798us) server_negotiation.cc:407] Connection header received
0618 22:31:00.813768 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:00.813771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:00.813821 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:00.813896 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:00.815173 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.815706 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.816877 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.817143 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.817228 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:00.819987 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:00.819990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:00.819992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:00.820015 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:00.821563 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.822094 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.822097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.822099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.822195 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:00.822677 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:00.822681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:00.822683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:00.822914 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:00.823018 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:00.823583 (+ 565us) server_negotiation.cc:299] Negotiation successful
0618 22:31:00.823726 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:31:00.824764 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:00.811834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37547 (local address 127.16.243.130:39593)
0618 22:31:00.812286 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:00.812331 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:00.813565 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:00.814102 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:00.814109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:00.814469 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:00.815023 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:00.815037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.815805 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.815807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:00.816749 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:00.816759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:00.817273 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:00.817275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:00.817417 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:00.818179 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:00.818200 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:00.819816 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:00.821702 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:00.821708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:00.821723 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:00.821991 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:00.822311 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:00.822314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:00.822315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:00.822536 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:00.823036 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:00.823042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:00.823325 (+ 283us) client_negotiation.cc:769] Sending connection context
0618 22:31:00.823493 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:31:00.823828 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260618 22:31:00.826165 4418 heartbeater.cc:347] Connected to a master server at 127.16.243.190:37547
I20260618 22:31:00.826403 4418 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:00.826969 4418 heartbeater.cc:511] Master 127.16.243.190:37547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.243.131@KRBTEST.COM; defaulting to no policy
I20260618 22:31:00.828033 4093 ts_manager.cc:194] Registered new tserver with Master: bef9527e3277446188d02fdafccaf701 (127.16.243.130:39333)
I20260618 22:31:00.828660 4093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.130@KRBTEST.COM'} at 127.16.243.130:39593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
Entry for principal kudu/127.16.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab.
I20260618 22:31:00.843873 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:00.953807 4423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:00.954085 4423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:00.954138 4423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:00.958225 4423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:00.958305 4423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:00.958371 4423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:31:00.963804 4423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.131.keytab
--principal=kudu/127.16.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:37547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:00.964879 4423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:00.965894 4423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:00.975448 4431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:00.975731 4423 server_base.cc:1034] running on GCE node
W20260618 22:31:00.975455 4428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:00.975455 4429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:00.976276 4423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:00.976938 4423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:00.978104 4423 hybrid_clock.cc:648] HybridClock initialized: now 1781821860978083 us; error 32 us; skew 500 ppm
Jun 18 22:31:00 dist-test-slave-jm5g krb5kdc[4048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:00.980829 4423 init.cc:377] Logged in from keytab as kudu/127.16.243.131@KRBTEST.COM (short username kudu)
I20260618 22:31:00.981695 4423 webserver.cc:460] Webserver started at http://127.16.243.131:40433/ using document root <none> and password file <none>
I20260618 22:31:00.982214 4423 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:00.982270 4423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:00.982437 4423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:00.984573 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "9ae7bb8afa81445588bbebf2fe3559f9"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "fc1c36b98dcf09ad17941fe82e65e52c"
server_key_iv: "b60604ecc1aebe0672645c45d53e48f2"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.985085 4423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "9ae7bb8afa81445588bbebf2fe3559f9"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "fc1c36b98dcf09ad17941fe82e65e52c"
server_key_iv: "b60604ecc1aebe0672645c45d53e48f2"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.988885 4423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:31:00.991362 4438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:00.992686 4423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:00.992790 4423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "9ae7bb8afa81445588bbebf2fe3559f9"
format_stamp: "Formatted at 2026-06-18 22:31:00 on dist-test-slave-jm5g"
server_key: "fc1c36b98dcf09ad17941fe82e65e52c"
server_key_iv: "b60604ecc1aebe0672645c45d53e48f2"
server_key_version: "encryptionkey@0"
I20260618 22:31:00.992906 4423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:01.010329 4423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:01.013396 4423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:01.013587 4423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:01.014178 4423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:01.015049 4423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:01.015096 4423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.015134 4423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:01.015148 4423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.023523 4423 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:33407
I20260618 22:31:01.023540 4551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:33407 every 8 connection(s)
I20260618 22:31:01.024427 4423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:31:01.029939 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4423
I20260618 22:31:01.030086 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:31:01.030380 17358 external_mini_cluster.cc:1428] Setting key d6361c93a7e523873dbe35c2044fcf06
Jun 18 22:31:01 dist-test-slave-jm5g krb5kdc[4048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.131@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:01.038631 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.026306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:45275 (local address 127.16.243.190:37547)
0618 22:31:01.026488 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.026492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.027392 (+ 900us) server_negotiation.cc:407] Connection header received
0618 22:31:01.028866 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.028869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.028921 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.028987 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:01.030616 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.031278 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.032454 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.032672 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.032757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.035288 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:01.035291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:01.035293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:01.035320 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:01.036685 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:01.037209 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:01.037212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:01.037213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:01.037294 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:01.037594 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:01.037597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:01.037598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:01.037788 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:01.037866 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.038347 (+ 481us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.038457 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 22:31:01.039532 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.026744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37547 (local address 127.16.243.131:45275)
0618 22:31:01.027236 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.027280 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.028629 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.029189 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.029197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.029633 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:01.030403 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.030426 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.031399 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.031402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.032338 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.032348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.032814 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.032818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.032982 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.033550 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:01.033568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:01.035093 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:01.036799 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:01.036805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:01.036820 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:01.037114 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:01.037364 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:01.037366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:01.037367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:01.037512 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:01.037872 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:01.037878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:01.038151 (+ 273us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.038314 (+ 163us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.038647 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260618 22:31:01.040690 4552 heartbeater.cc:347] Connected to a master server at 127.16.243.190:37547
I20260618 22:31:01.040911 4552 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:01.041385 4552 heartbeater.cc:511] Master 127.16.243.190:37547 requested a full tablet report, sending...
I20260618 22:31:01.042402 4093 ts_manager.cc:194] Registered new tserver with Master: 9ae7bb8afa81445588bbebf2fe3559f9 (127.16.243.131:33407)
I20260618 22:31:01.043100 4093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.243.131@KRBTEST.COM'} at 127.16.243.131:45275
I20260618 22:31:01.044227 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:31:01.053123 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.046122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.16.243.190:37547)
0618 22:31:01.046249 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.046254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.046388 (+ 134us) server_negotiation.cc:407] Connection header received
0618 22:31:01.046511 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.046514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.046584 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.046651 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:01.047346 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.047932 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.048743 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.048955 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.049048 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.050331 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:01.050334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:01.050336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:01.050361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:01.051552 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:01.051926 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:01.051930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:01.051932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:01.052008 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:01.052262 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:01.052265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:01.052267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:01.052487 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:01.052605 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.052848 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.052958 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:31:01.055979 4093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:31:01.059008 4093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:31:01.073787 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.067889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.16.243.131:33407)
0618 22:31:01.069053 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.069065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.069082 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:31:01.069143 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.069148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.069327 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.069465 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:01.069708 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.070485 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.072094 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.072966 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073288 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073399 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.073496 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.073644 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":100,"threads_started":1}
I20260618 22:31:01.073813 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.067314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44159 (local address 127.0.0.1:58742)
0618 22:31:01.067919 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.067970 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.068128 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.068856 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.068861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.068884 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:01.069173 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.069179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.070568 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.070572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.072051 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.072062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.073283 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.073361 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073383 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.073458 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.073593 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":155,"thread_start_us":91,"threads_started":1}
I20260618 22:31:01.074047 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.068270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.16.243.130:39333)
0618 22:31:01.068593 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.068605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.069042 (+ 437us) server_negotiation.cc:407] Connection header received
0618 22:31:01.069145 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.069149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.069328 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.069464 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:01.069851 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.070966 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.072355 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.072937 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073054 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073207 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.073736 (+ 529us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.073826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":143,"threads_started":1}
I20260618 22:31:01.074198 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.067534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:33407 (local address 127.0.0.1:50166)
0618 22:31:01.068309 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.068323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.068404 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.069467 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.069470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.069489 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:01.069594 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.069600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.070639 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.070643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.071978 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.071987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.073101 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.073300 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073341 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.073449 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.073593 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":156,"thread_start_us":115,"threads_started":1}
I20260618 22:31:01.074308 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.067422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.16.243.129:44159)
0618 22:31:01.068355 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.068371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.068395 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:31:01.068470 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.068476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.068700 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.068868 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:01.069288 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.070398 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.072159 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.072971 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073100 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073230 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.074029 (+ 799us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.074172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":89,"threads_started":1}
I20260618 22:31:01.074581 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.068069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:39333 (local address 127.0.0.1:45568)
0618 22:31:01.068728 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.068740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.068832 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.069628 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.069630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.069645 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:01.069732 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.069738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.071106 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.071110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.072248 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.072255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.073053 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.073056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.073224 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.073338 (+ 114us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.073449 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.073593 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":88,"thread_start_us":75,"threads_started":1}
I20260618 22:31:01.077091 4486 tablet_service.cc:1467] Processing CreateTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 (DEFAULT_TABLE table=test-table [id=fc244258e43f40cd9f4dc916476f3efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:01.077100 4352 tablet_service.cc:1467] Processing CreateTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 (DEFAULT_TABLE table=test-table [id=fc244258e43f40cd9f4dc916476f3efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:01.078089 4352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d56df41d4c74bd18f16b4fdfb192c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:01.078088 4486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d56df41d4c74bd18f16b4fdfb192c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:01.084712 4570 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Bootstrap starting.
I20260618 22:31:01.085157 4569 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Bootstrap starting.
I20260618 22:31:01.086841 4570 tablet_bootstrap.cc:654] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:01.086764 4218 tablet_service.cc:1467] Processing CreateTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 (DEFAULT_TABLE table=test-table [id=fc244258e43f40cd9f4dc916476f3efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:01.087606 4569 tablet_bootstrap.cc:654] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:01.087981 4218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d56df41d4c74bd18f16b4fdfb192c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:01.088085 4570 log.cc:824] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:01.088852 4569 log.cc:824] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:01.090090 4570 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: No bootstrap required, opened a new log
I20260618 22:31:01.090289 4570 ts_tablet_manager.cc:1397] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:31:01.090824 4569 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: No bootstrap required, opened a new log
I20260618 22:31:01.091004 4569 ts_tablet_manager.cc:1397] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:31:01.093308 4570 raft_consensus.cc:348] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.093550 4570 raft_consensus.cc:374] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:01.093602 4570 raft_consensus.cc:729] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae7bb8afa81445588bbebf2fe3559f9, State: Initialized, Role: FOLLOWER
I20260618 22:31:01.094039 4570 consensus_queue.cc:260] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.094739 4552 heartbeater.cc:503] Master 127.16.243.190:37547 was elected leader, sending a full tablet report...
I20260618 22:31:01.094767 4569 raft_consensus.cc:348] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.095005 4569 raft_consensus.cc:374] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:01.095038 4570 ts_tablet_manager.cc:1428] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:31:01.095057 4569 raft_consensus.cc:729] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef9527e3277446188d02fdafccaf701, State: Initialized, Role: FOLLOWER
I20260618 22:31:01.095467 4569 consensus_queue.cc:260] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.096191 4418 heartbeater.cc:503] Master 127.16.243.190:37547 was elected leader, sending a full tablet report...
I20260618 22:31:01.096467 4569 ts_tablet_manager.cc:1428] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:31:01.096872 4575 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: Bootstrap starting.
I20260618 22:31:01.099381 4575 tablet_bootstrap.cc:654] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:01.100687 4575 log.cc:824] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:01.102702 4575 tablet_bootstrap.cc:492] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: No bootstrap required, opened a new log
I20260618 22:31:01.102906 4575 ts_tablet_manager.cc:1397] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:31:01.105887 4575 raft_consensus.cc:348] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.106138 4575 raft_consensus.cc:374] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:01.106173 4575 raft_consensus.cc:729] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8946884e9342e4a312c8b1714f3081, State: Initialized, Role: FOLLOWER
I20260618 22:31:01.106688 4575 consensus_queue.cc:260] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.107477 4284 heartbeater.cc:503] Master 127.16.243.190:37547 was elected leader, sending a full tablet report...
I20260618 22:31:01.107842 4575 ts_tablet_manager.cc:1428] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:31:01.267560 4573 raft_consensus.cc:482] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:31:01.267786 4573 raft_consensus.cc:504] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.269007 4573 leader_election.cc:288] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8946884e9342e4a312c8b1714f3081 (127.16.243.129:44159), bef9527e3277446188d02fdafccaf701 (127.16.243.130:39333)
I20260618 22:31:01.273866 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.269209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44159 (local address 127.16.243.131:59317)
0618 22:31:01.269417 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.269432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.269551 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.269829 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.269830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.269846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:01.269985 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.269993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.271043 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.271047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.272579 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.272591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.273411 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.273414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.273565 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.273582 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.273644 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.273721 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":130}
I20260618 22:31:01.273881 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.269327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:59317 (local address 127.16.243.129:44159)
0618 22:31:01.269437 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.269441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.269542 (+ 101us) server_negotiation.cc:407] Connection header received
0618 22:31:01.269667 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.269670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.269731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.269814 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:01.270112 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.270923 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.272701 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.273297 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.273411 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.273479 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.273666 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.273742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:31:01.273893 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.269358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:51543 (local address 127.16.243.130:39333)
0618 22:31:01.269492 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.269496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.269673 (+ 177us) server_negotiation.cc:407] Connection header received
0618 22:31:01.269750 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.269753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.269800 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.269864 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:01.270115 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.270923 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.272698 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.273293 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.273411 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.273462 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.273725 (+ 263us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.273794 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:31:01.273903 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.269269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:39333 (local address 127.16.243.131:51543)
0618 22:31:01.269581 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.269592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.269678 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.269885 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.269888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.269899 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:01.270011 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.270017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.271043 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.271047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.272588 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.272594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.273411 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.273414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.273538 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.273564 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.273697 (+ 133us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.273748 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":177,"thread_start_us":129,"threads_started":1}
I20260618 22:31:01.274564 4238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d56df41d4c74bd18f16b4fdfb192c81" candidate_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8946884e9342e4a312c8b1714f3081" is_pre_election: true
I20260618 22:31:01.274544 4372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d56df41d4c74bd18f16b4fdfb192c81" candidate_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef9527e3277446188d02fdafccaf701" is_pre_election: true
I20260618 22:31:01.274892 4372 raft_consensus.cc:2454] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae7bb8afa81445588bbebf2fe3559f9 in term 0.
I20260618 22:31:01.274892 4238 raft_consensus.cc:2454] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ae7bb8afa81445588bbebf2fe3559f9 in term 0.
W20260618 22:31:01.275339 4553 tablet.cc:2369] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:31:01.275410 4439 leader_election.cc:302] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ae7bb8afa81445588bbebf2fe3559f9, bef9527e3277446188d02fdafccaf701; no voters:
I20260618 22:31:01.275681 4573 raft_consensus.cc:2781] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:31:01.275758 4573 raft_consensus.cc:482] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:31:01.275822 4573 raft_consensus.cc:3037] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:01.276808 4573 raft_consensus.cc:504] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.277127 4573 leader_election.cc:288] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e8946884e9342e4a312c8b1714f3081 (127.16.243.129:44159), bef9527e3277446188d02fdafccaf701 (127.16.243.130:39333)
I20260618 22:31:01.277474 4238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d56df41d4c74bd18f16b4fdfb192c81" candidate_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8946884e9342e4a312c8b1714f3081"
I20260618 22:31:01.277465 4372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d56df41d4c74bd18f16b4fdfb192c81" candidate_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef9527e3277446188d02fdafccaf701"
I20260618 22:31:01.277583 4372 raft_consensus.cc:3037] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:01.277609 4238 raft_consensus.cc:3037] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:01.278635 4238 raft_consensus.cc:2454] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae7bb8afa81445588bbebf2fe3559f9 in term 1.
I20260618 22:31:01.278647 4372 raft_consensus.cc:2454] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ae7bb8afa81445588bbebf2fe3559f9 in term 1.
I20260618 22:31:01.278944 4442 leader_election.cc:302] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8946884e9342e4a312c8b1714f3081, 9ae7bb8afa81445588bbebf2fe3559f9; no voters:
I20260618 22:31:01.279111 4573 raft_consensus.cc:2781] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:01.279289 4573 raft_consensus.cc:686] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae7bb8afa81445588bbebf2fe3559f9, State: Running, Role: LEADER
I20260618 22:31:01.279633 4573 consensus_queue.cc:237] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } }
I20260618 22:31:01.282660 4091 catalog_manager.cc:5495] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ae7bb8afa81445588bbebf2fe3559f9 (127.16.243.131). New cstate: current_term: 1 leader_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae7bb8afa81445588bbebf2fe3559f9" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 } health_report { overall_health: UNKNOWN } } }
W20260618 22:31:01.310724 4419 tablet.cc:2369] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:31:01.311605 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.307748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.16.243.131:33407)
0618 22:31:01.307858 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.307862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.307874 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:31:01.307947 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.307949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.308005 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.308093 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:01.308356 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.308858 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.310063 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.310278 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.310352 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.310695 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:01.311209 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:01.311331 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.311408 (+ 77us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.311464 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:31:01.317700 4238 raft_consensus.cc:1261] T 8d56df41d4c74bd18f16b4fdfb192c81 P 1e8946884e9342e4a312c8b1714f3081 [term 1 FOLLOWER]: Refusing update from remote peer 9ae7bb8afa81445588bbebf2fe3559f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:01.317703 4372 raft_consensus.cc:1261] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 1 FOLLOWER]: Refusing update from remote peer 9ae7bb8afa81445588bbebf2fe3559f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:01.318388 4573 consensus_queue.cc:1035] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8946884e9342e4a312c8b1714f3081" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 44159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:01.318615 4579 consensus_queue.cc:1035] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef9527e3277446188d02fdafccaf701" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:01.335111 4093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:31:01.335276 4093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:31:01.337604 4093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f7f1194ddf4102922be6ce1d995fe5: Sending DeleteTablet for 3 replicas of tablet 8d56df41d4c74bd18f16b4fdfb192c81
I20260618 22:31:01.338451 4352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:01 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:45568
I20260618 22:31:01.338447 4218 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:01 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:58742
I20260618 22:31:01.339159 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4155
I20260618 22:31:01.339295 4592 tablet_replica.cc:330] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: stopping tablet replica
I20260618 22:31:01.339299 4486 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d56df41d4c74bd18f16b4fdfb192c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:01 UTC) from {username='kudu', principal='kudu/127.16.243.190@KRBTEST.COM'} at 127.0.0.1:50166
I20260618 22:31:01.339684 4592 raft_consensus.cc:2229] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:31:01.340006 4592 raft_consensus.cc:2258] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:31:01.340534 4594 tablet_replica.cc:330] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: stopping tablet replica
I20260618 22:31:01.340816 4594 raft_consensus.cc:2229] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:31:01.341159 4594 raft_consensus.cc:2258] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:31:01.342536 4594 ts_tablet_manager.cc:1905] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:31:01.343202 4592 ts_tablet_manager.cc:1905] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:31:01.345165 4592 ts_tablet_manager.cc:1918] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:31:01.345253 4592 log.cc:1196] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/8d56df41d4c74bd18f16b4fdfb192c81
I20260618 22:31:01.345295 4594 ts_tablet_manager.cc:1918] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:31:01.345386 4594 log.cc:1196] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/8d56df41d4c74bd18f16b4fdfb192c81
I20260618 22:31:01.345534 4592 ts_tablet_manager.cc:1939] T 8d56df41d4c74bd18f16b4fdfb192c81 P bef9527e3277446188d02fdafccaf701: Deleting consensus metadata
I20260618 22:31:01.345700 4594 ts_tablet_manager.cc:1939] T 8d56df41d4c74bd18f16b4fdfb192c81 P 9ae7bb8afa81445588bbebf2fe3559f9: Deleting consensus metadata
I20260618 22:31:01.346836 4077 catalog_manager.cc:4841] TS 9ae7bb8afa81445588bbebf2fe3559f9 (127.16.243.131:33407): tablet 8d56df41d4c74bd18f16b4fdfb192c81 (table test-table [id=fc244258e43f40cd9f4dc916476f3efa]) successfully deleted
I20260618 22:31:01.347049 4077 catalog_manager.cc:4841] TS bef9527e3277446188d02fdafccaf701 (127.16.243.130:39333): tablet 8d56df41d4c74bd18f16b4fdfb192c81 (table test-table [id=fc244258e43f40cd9f4dc916476f3efa]) successfully deleted
W20260618 22:31:01.349364 4080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:44159 (error 108)
I20260618 22:31:01.349412 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4289
I20260618 22:31:01.350646 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.350345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:44159 (local address 127.0.0.1:58758)
0618 22:31:01.350466 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.350530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:44159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 22:31:01.350847 4080 catalog_manager.cc:4584] TS 1e8946884e9342e4a312c8b1714f3081 (127.16.243.129:44159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d56df41d4c74bd18f16b4fdfb192c81: Network error: Client connection negotiation failed: client connection to 127.16.243.129:44159: connect: Connection refused (error 111)
I20260618 22:31:01.356235 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4423
I20260618 22:31:01.360953 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4061
2026-06-18T22:31:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2313 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-18T22:31:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:01Z Disabled control of system clock
I20260618 22:31:01.388805 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35459
--webserver_interface=127.16.243.190
--webserver_port=0
--builtin_ntp_servers=127.16.243.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:31:01.496575 4599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:01.496863 4599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:01.496901 4599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:01.500967 4599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:01.501041 4599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:01.501055 4599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:01.501071 4599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:01.501085 4599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:01.505940 4599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:35459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:01.507051 4599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:01.508388 4599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:01.516716 4605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:01.516729 4604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:01.516700 4607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:01.519582 4599 server_base.cc:1034] running on GCE node
I20260618 22:31:01.520313 4599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:01.521733 4599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:01.522893 4599 hybrid_clock.cc:648] HybridClock initialized: now 1781821861522866 us; error 42 us; skew 500 ppm
I20260618 22:31:01.524895 4599 webserver.cc:460] Webserver started at http://127.16.243.190:42997/ using document root <none> and password file <none>
I20260618 22:31:01.525446 4599 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:01.525511 4599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:01.525696 4599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:01.527773 4599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "b63cce56511e4630922dcbdc5a8a4417"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "887804cddd37663d1c87c5b141396bba"
server_key_iv: "144862f10541686eecf9b57a58707248"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.528294 4599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "b63cce56511e4630922dcbdc5a8a4417"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "887804cddd37663d1c87c5b141396bba"
server_key_iv: "144862f10541686eecf9b57a58707248"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.531997 4599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:31:01.534407 4613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.535651 4599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:01.535755 4599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "b63cce56511e4630922dcbdc5a8a4417"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "887804cddd37663d1c87c5b141396bba"
server_key_iv: "144862f10541686eecf9b57a58707248"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.535864 4599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:01.546155 4599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:01.546854 4599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:01.547027 4599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:01.553443 4599 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:35459
I20260618 22:31:01.553449 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:35459 every 8 connection(s)
I20260618 22:31:01.554397 4599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:01.557343 4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:01.562705 4666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: Bootstrap starting.
I20260618 22:31:01.564191 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4599
I20260618 22:31:01.564303 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:01.564571 17358 external_mini_cluster.cc:1428] Setting key a2522ee7f71d4c1736adef9b6b134190
I20260618 22:31:01.565208 4666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:01.565974 4666 log.cc:824] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:01.567800 4666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: No bootstrap required, opened a new log
I20260618 22:31:01.571218 4666 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } }
I20260618 22:31:01.571444 4666 raft_consensus.cc:374] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:01.571498 4666 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63cce56511e4630922dcbdc5a8a4417, State: Initialized, Role: FOLLOWER
I20260618 22:31:01.571899 4666 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } }
I20260618 22:31:01.572022 4666 raft_consensus.cc:388] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:01.572081 4666 raft_consensus.cc:482] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:01.572170 4666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:01.573035 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.565616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.16.243.190:35459)
0618 22:31:01.566288 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.566309 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.566349 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:31:01.567017 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.567036 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.567363 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.567665 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:01.568075 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.569009 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.569839 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.570304 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.570443 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.571264 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:01.571269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:01.571282 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:01.571719 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:01.571789 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:01.571811 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:01.571900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.572031 (+ 131us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.572310 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260618 22:31:01.573041 4666 raft_consensus.cc:504] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } }
I20260618 22:31:01.573416 4666 leader_election.cc:302] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b63cce56511e4630922dcbdc5a8a4417; no voters:
I20260618 22:31:01.573755 4666 leader_election.cc:288] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:01.573865 4671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:01.574071 4671 raft_consensus.cc:686] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [term 1 LEADER]: Becoming Leader. State: Replica: b63cce56511e4630922dcbdc5a8a4417, State: Running, Role: LEADER
I20260618 22:31:01.574362 4671 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } }
I20260618 22:31:01.574982 4666 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:01.575795 4672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b63cce56511e4630922dcbdc5a8a4417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } } }
I20260618 22:31:01.575923 4672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:01.576269 4680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:01.576236 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63cce56511e4630922dcbdc5a8a4417. Latest consensus state: current_term: 1 leader_uuid: "b63cce56511e4630922dcbdc5a8a4417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63cce56511e4630922dcbdc5a8a4417" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35459 } } }
I20260618 22:31:01.576368 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:01.579177 4680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:01.585359 4680 catalog_manager.cc:1347] Generated new cluster ID: 071dc86b2f374b049dfcd82b75152991
I20260618 22:31:01.585466 4680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:01.595913 4680 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:01.596851 4680 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:01.602730 4680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: Generated new TSK 0
I20260618 22:31:01.603457 4680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:01.610879 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.129:0
--local_ip_for_outbound_sockets=127.16.243.129
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--builtin_ntp_servers=127.16.243.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:31:01.718508 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:01.718823 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:01.718865 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:01.722868 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:01.722924 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:01.722981 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.129
I20260618 22:31:01.727741 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.243.129
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.129
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:01.728782 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:01.730154 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:01.739293 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:01.739310 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:01.739300 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:01.739518 4690 server_base.cc:1034] running on GCE node
I20260618 22:31:01.740000 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:01.740630 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:01.741786 4690 hybrid_clock.cc:648] HybridClock initialized: now 1781821861741768 us; error 31 us; skew 500 ppm
I20260618 22:31:01.743726 4690 webserver.cc:460] Webserver started at http://127.16.243.129:42341/ using document root <none> and password file <none>
I20260618 22:31:01.744287 4690 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:01.744349 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:01.744534 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:01.746521 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data/instance:
uuid: "624e4decb7b74175a01be3be27dad992"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "912bc879f4b480fb4432a01ac2cebdea"
server_key_iv: "160f683b4fc8bdc8b2b66a11e2d34141"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.747099 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance:
uuid: "624e4decb7b74175a01be3be27dad992"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "912bc879f4b480fb4432a01ac2cebdea"
server_key_iv: "160f683b4fc8bdc8b2b66a11e2d34141"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.750720 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:31:01.753067 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.754235 4690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:01.754340 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal
uuid: "624e4decb7b74175a01be3be27dad992"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "912bc879f4b480fb4432a01ac2cebdea"
server_key_iv: "160f683b4fc8bdc8b2b66a11e2d34141"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.754431 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:01.775383 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:01.776087 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:01.776268 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:01.776841 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:01.777768 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:01.777813 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.777864 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:01.777892 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.786298 4690 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.129:35339
I20260618 22:31:01.786373 4817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.129:35339 every 8 connection(s)
I20260618 22:31:01.787206 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/data/info.pb
I20260618 22:31:01.795850 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.788907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.129:52091 (local address 127.16.243.190:35459)
0618 22:31:01.789060 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.789064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.790080 (+ 1016us) server_negotiation.cc:407] Connection header received
0618 22:31:01.791377 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.791380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.791425 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.791496 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:01.792308 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.792846 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.793971 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.794186 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.794257 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.795459 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:01.795463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:01.795465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:01.795477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:01.795496 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:01.795503 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:01.795559 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.795676 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.795733 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:31:01.796697 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.789292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35459 (local address 127.16.243.129:52091)
0618 22:31:01.789924 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.789975 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.791200 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.791642 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.791648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.791729 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:01.792207 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.792218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.792971 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.792974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.793868 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.793885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.794341 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.794344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.794503 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.795199 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:31:01.795200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:01.795241 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:31:01.795307 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:01.795570 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:01.795575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:01.795579 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.795675 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.795907 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260618 22:31:01.796921 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4690
I20260618 22:31:01.797041 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-0/wal/instance
I20260618 22:31:01.797360 17358 external_mini_cluster.cc:1428] Setting key bb01e253de9eaad16e188a30e8e497c0
I20260618 22:31:01.798027 4818 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35459
I20260618 22:31:01.798264 4818 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:01.798877 4818 heartbeater.cc:511] Master 127.16.243.190:35459 requested a full tablet report, sending...
I20260618 22:31:01.799324 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.130:0
--local_ip_for_outbound_sockets=127.16.243.130
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--builtin_ntp_servers=127.16.243.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:31:01.800444 4630 ts_manager.cc:194] Registered new tserver with Master: 624e4decb7b74175a01be3be27dad992 (127.16.243.129:35339)
I20260618 22:31:01.801640 4630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.129:52091
W20260618 22:31:01.905645 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:01.906008 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:01.906075 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:01.910501 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:01.910598 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:01.910679 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.130
I20260618 22:31:01.915565 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.243.130
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.130
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:01.916646 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:01.918006 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:01.927161 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:01.927201 4821 server_base.cc:1034] running on GCE node
W20260618 22:31:01.927165 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:01.927165 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:01.927835 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:01.928470 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:01.929638 4821 hybrid_clock.cc:648] HybridClock initialized: now 1781821861929607 us; error 43 us; skew 500 ppm
I20260618 22:31:01.931468 4821 webserver.cc:460] Webserver started at http://127.16.243.130:40017/ using document root <none> and password file <none>
I20260618 22:31:01.932063 4821 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:01.932126 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:01.932320 4821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:01.934336 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data/instance:
uuid: "5173162f82364e3aa217fa63752a2266"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "80dd6f3c10cfd2b249ec352b91ae9b8d"
server_key_iv: "daffca5f2635e36ef6b1320713b2f5b0"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.934945 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance:
uuid: "5173162f82364e3aa217fa63752a2266"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "80dd6f3c10cfd2b249ec352b91ae9b8d"
server_key_iv: "daffca5f2635e36ef6b1320713b2f5b0"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.938475 4821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:01.940932 4835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.942103 4821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:01.942198 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal
uuid: "5173162f82364e3aa217fa63752a2266"
format_stamp: "Formatted at 2026-06-18 22:31:01 on dist-test-slave-jm5g"
server_key: "80dd6f3c10cfd2b249ec352b91ae9b8d"
server_key_iv: "daffca5f2635e36ef6b1320713b2f5b0"
server_key_version: "encryptionkey@0"
I20260618 22:31:01.942281 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:01.964968 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:01.965703 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:01.965852 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:01.966421 4821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:01.967367 4821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:01.967410 4821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.967449 4821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:01.967465 4821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:01.975829 4821 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.130:36339
I20260618 22:31:01.975900 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.130:36339 every 8 connection(s)
I20260618 22:31:01.976779 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/data/info.pb
I20260618 22:31:01.985584 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.978512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.130:51343 (local address 127.16.243.190:35459)
0618 22:31:01.978673 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:01.978677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:01.979711 (+ 1034us) server_negotiation.cc:407] Connection header received
0618 22:31:01.981046 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:01.981050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:01.981118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:01.981191 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:01.982079 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.982644 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.983764 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.984009 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.984108 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.985131 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:01.985135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:01.985138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:01.985152 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:01.985171 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:01.985178 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:01.985234 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:01.985368 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:31:01.985433 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 22:31:01.985839 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4821
I20260618 22:31:01.985960 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal/instance
I20260618 22:31:01.986225 17358 external_mini_cluster.cc:1428] Setting key aaf745163ae5f89863c61f01bb84b1a7
I20260618 22:31:01.986577 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:01.978938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35459 (local address 127.16.243.130:51343)
0618 22:31:01.979563 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:01.979611 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:01.980848 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:01.981365 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:01.981372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:01.981456 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:01.981940 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.981955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.982756 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.982759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.983653 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:01.983662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:01.984154 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:01.984158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:01.984317 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:01.984879 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:31:01.984881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:01.984922 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:31:01.984985 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:01.985243 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:01.985253 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:01.985258 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:31:01.985361 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 22:31:01.985651 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260618 22:31:01.987733 4949 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35459
I20260618 22:31:01.987979 4949 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:01.988169 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.243.131:0
--local_ip_for_outbound_sockets=127.16.243.131
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--builtin_ntp_servers=127.16.243.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:31:01.988590 4949 heartbeater.cc:511] Master 127.16.243.190:35459 requested a full tablet report, sending...
I20260618 22:31:01.989681 4630 ts_manager.cc:194] Registered new tserver with Master: 5173162f82364e3aa217fa63752a2266 (127.16.243.130:36339)
I20260618 22:31:01.990420 4630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.130:51343
W20260618 22:31:02.097056 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:02.097334 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:02.097373 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:02.101485 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:02.101552 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:02.101613 4952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.243.131
I20260618 22:31:02.106580 4952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.243.131
--webserver_port=0
--tserver_master_addrs=127.16.243.190:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.243.131
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:02.107658 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:02.108986 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:02.118284 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:02.118428 4952 server_base.cc:1034] running on GCE node
W20260618 22:31:02.118288 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:02.118647 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:02.119035 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:02.119719 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:02.120894 4952 hybrid_clock.cc:648] HybridClock initialized: now 1781821862120868 us; error 41 us; skew 500 ppm
I20260618 22:31:02.122822 4952 webserver.cc:460] Webserver started at http://127.16.243.131:45133/ using document root <none> and password file <none>
I20260618 22:31:02.123440 4952 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:02.123509 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:02.123701 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:02.125837 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data/instance:
uuid: "d856c8fbe84b404c8159de0bdf3681ef"
format_stamp: "Formatted at 2026-06-18 22:31:02 on dist-test-slave-jm5g"
server_key: "f808f99a229ce78053295ac4631ec274"
server_key_iv: "ec616057c4059d41c2239c6fc1cead2d"
server_key_version: "encryptionkey@0"
I20260618 22:31:02.126403 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance:
uuid: "d856c8fbe84b404c8159de0bdf3681ef"
format_stamp: "Formatted at 2026-06-18 22:31:02 on dist-test-slave-jm5g"
server_key: "f808f99a229ce78053295ac4631ec274"
server_key_iv: "ec616057c4059d41c2239c6fc1cead2d"
server_key_version: "encryptionkey@0"
I20260618 22:31:02.130239 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260618 22:31:02.132988 4966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:02.134341 4952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:31:02.134454 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal
uuid: "d856c8fbe84b404c8159de0bdf3681ef"
format_stamp: "Formatted at 2026-06-18 22:31:02 on dist-test-slave-jm5g"
server_key: "f808f99a229ce78053295ac4631ec274"
server_key_iv: "ec616057c4059d41c2239c6fc1cead2d"
server_key_version: "encryptionkey@0"
I20260618 22:31:02.134610 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:02.159935 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:02.160657 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:02.160835 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:02.161445 4952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:31:02.162412 4952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:31:02.162458 4952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:02.162497 4952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:31:02.162521 4952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:02.171059 4952 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.131:36951
I20260618 22:31:02.171110 5079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.131:36951 every 8 connection(s)
I20260618 22:31:02.171959 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/data/info.pb
I20260618 22:31:02.174821 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 4952
I20260618 22:31:02.174934 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal/instance
I20260618 22:31:02.175216 17358 external_mini_cluster.cc:1428] Setting key d222d3b008b6cdaa790370ee4934e85e
I20260618 22:31:02.181411 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.173694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:50215 (local address 127.16.243.190:35459)
0618 22:31:02.173824 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.173828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.174997 (+ 1169us) server_negotiation.cc:407] Connection header received
0618 22:31:02.176890 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.176893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.176971 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.177040 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:02.177953 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.178470 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.179614 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.179832 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.179918 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.180954 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:02.180957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:02.180959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:02.180972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:02.180991 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:02.180998 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:02.181053 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.181198 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.181262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:31:02.182281 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.174090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:35459 (local address 127.16.243.131:50215)
0618 22:31:02.174841 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.174888 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.176683 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.177231 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.177238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.177329 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:02.177814 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.177827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.178605 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.178608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.179495 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.179504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.179968 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.179972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.180143 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.180701 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:31:02.180702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:02.180744 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:31:02.180812 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:02.181062 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:02.181079 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:02.181085 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.181192 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.181486 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260618 22:31:02.183377 5080 heartbeater.cc:347] Connected to a master server at 127.16.243.190:35459
I20260618 22:31:02.183589 5080 heartbeater.cc:464] Registering TS with master...
I20260618 22:31:02.184087 5080 heartbeater.cc:511] Master 127.16.243.190:35459 requested a full tablet report, sending...
I20260618 22:31:02.185073 4630 ts_manager.cc:194] Registered new tserver with Master: d856c8fbe84b404c8159de0bdf3681ef (127.16.243.131:36951)
I20260618 22:31:02.185642 4630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.243.131:50215
I20260618 22:31:02.189005 17358 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:31:02.194710 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.190929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.16.243.190:35459)
0618 22:31:02.191065 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.191069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.191257 (+ 188us) server_negotiation.cc:407] Connection header received
0618 22:31:02.191324 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.191326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.191368 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.191456 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:02.191680 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.192186 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.193032 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.193263 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.193349 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.194193 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:02.194196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:02.194198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:02.194211 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:02.194231 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:02.194239 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:02.194299 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.194465 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.194533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:31:02.197566 4630 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:31:02.200655 4630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:31:02.214783 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.209434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.131:36951 (local address 127.0.0.1:45060)
0618 22:31:02.210465 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.210476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.210580 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.211056 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.211058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.211087 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:02.211163 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.211168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.212433 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.212436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.213480 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.213487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214205 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.214314 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214351 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.214461 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.214627 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260618 22:31:02.214787 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.209850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.16.243.131:36951)
0618 22:31:02.210294 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.210305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.210592 (+ 287us) server_negotiation.cc:407] Connection header received
0618 22:31:02.210670 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.210675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.210858 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.210999 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:02.211263 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.212319 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.213578 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214096 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214206 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214389 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.214496 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.214627 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":155,"threads_started":1}
I20260618 22:31:02.215020 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.208865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35339 (local address 127.0.0.1:51748)
0618 22:31:02.209306 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.209356 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.209479 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.210327 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.210333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.210357 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:02.210492 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.210499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.211918 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.211921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.213197 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.213207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214451 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.214687 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214720 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.214789 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.214871 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":115,"thread_start_us":75,"threads_started":1}
I20260618 22:31:02.215461 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.208997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.16.243.129:35339)
0618 22:31:02.209655 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.209665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.209685 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:31:02.209751 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.209757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.209966 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.210138 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:02.210647 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.211806 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.213300 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214098 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214213 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214388 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.215190 (+ 802us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.215302 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":107,"threads_started":1}
I20260618 22:31:02.215610 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.208987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36339 (local address 127.0.0.1:52458)
0618 22:31:02.210047 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.210061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.210141 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.210824 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.210827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.210842 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:02.210938 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.210945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.211961 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.211964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.213154 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.213164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214451 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.214695 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214731 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.214788 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.215431 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":692,"thread_start_us":102,"threads_started":1}
I20260618 22:31:02.216114 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.209690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.16.243.130:36339)
0618 22:31:02.210368 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.210380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.210403 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:31:02.210474 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.210480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.210712 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.210838 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:02.211035 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.211835 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.213283 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.214099 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.214214 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.214388 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.215836 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.215945 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":106,"threads_started":1}
I20260618 22:31:02.217945 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 5167835f43fd4328ba7f7966e5373376 (DEFAULT_TABLE table=test-table [id=4d19a444d6ea4ab49d70566b3cf3c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:02.218138 4752 tablet_service.cc:1467] Processing CreateTablet for tablet 5167835f43fd4328ba7f7966e5373376 (DEFAULT_TABLE table=test-table [id=4d19a444d6ea4ab49d70566b3cf3c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:02.219113 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5167835f43fd4328ba7f7966e5373376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:02.219113 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5167835f43fd4328ba7f7966e5373376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:02.220670 4883 tablet_service.cc:1467] Processing CreateTablet for tablet 5167835f43fd4328ba7f7966e5373376 (DEFAULT_TABLE table=test-table [id=4d19a444d6ea4ab49d70566b3cf3c218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:31:02.221522 4883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5167835f43fd4328ba7f7966e5373376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:02.226445 5097 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Bootstrap starting.
I20260618 22:31:02.227054 5098 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Bootstrap starting.
I20260618 22:31:02.228440 5099 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Bootstrap starting.
I20260618 22:31:02.229087 5098 tablet_bootstrap.cc:654] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:02.229146 5097 tablet_bootstrap.cc:654] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:02.229786 5098 log.cc:824] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:02.229926 5097 log.cc:824] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:02.230389 5099 tablet_bootstrap.cc:654] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:02.231006 5099 log.cc:824] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:02.231670 5098 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: No bootstrap required, opened a new log
I20260618 22:31:02.231858 5098 ts_tablet_manager.cc:1397] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:31:02.232528 5097 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: No bootstrap required, opened a new log
I20260618 22:31:02.232601 5099 tablet_bootstrap.cc:492] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: No bootstrap required, opened a new log
I20260618 22:31:02.232709 5097 ts_tablet_manager.cc:1397] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:31:02.232762 5099 ts_tablet_manager.cc:1397] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260618 22:31:02.236011 5099 raft_consensus.cc:348] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.236035 5098 raft_consensus.cc:348] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.236244 5099 raft_consensus.cc:374] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:02.236282 5098 raft_consensus.cc:374] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:02.236303 5099 raft_consensus.cc:729] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5173162f82364e3aa217fa63752a2266, State: Initialized, Role: FOLLOWER
I20260618 22:31:02.236336 5098 raft_consensus.cc:729] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624e4decb7b74175a01be3be27dad992, State: Initialized, Role: FOLLOWER
I20260618 22:31:02.236712 5099 consensus_queue.cc:260] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.236639 5097 raft_consensus.cc:348] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.236843 5097 raft_consensus.cc:374] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:02.236788 5098 consensus_queue.cc:260] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.236892 5097 raft_consensus.cc:729] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d856c8fbe84b404c8159de0bdf3681ef, State: Initialized, Role: FOLLOWER
I20260618 22:31:02.237257 5097 consensus_queue.cc:260] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.237324 4949 heartbeater.cc:503] Master 127.16.243.190:35459 was elected leader, sending a full tablet report...
I20260618 22:31:02.237684 5099 ts_tablet_manager.cc:1428] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:31:02.237802 5098 ts_tablet_manager.cc:1428] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:31:02.238019 5097 ts_tablet_manager.cc:1428] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:31:02.238171 4818 heartbeater.cc:503] Master 127.16.243.190:35459 was elected leader, sending a full tablet report...
I20260618 22:31:02.238330 5080 heartbeater.cc:503] Master 127.16.243.190:35459 was elected leader, sending a full tablet report...
W20260618 22:31:02.288314 4819 tablet.cc:2369] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:31:02.295821 5105 raft_consensus.cc:482] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:31:02.296015 5105 raft_consensus.cc:504] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.297187 5105 leader_election.cc:288] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 624e4decb7b74175a01be3be27dad992 (127.16.243.129:35339), 5173162f82364e3aa217fa63752a2266 (127.16.243.130:36339)
I20260618 22:31:02.300938 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.297723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:54417 (local address 127.16.243.130:36339)
0618 22:31:02.297829 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.297833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.297965 (+ 132us) server_negotiation.cc:407] Connection header received
0618 22:31:02.298123 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.298127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.298178 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.298247 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:02.298484 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.299006 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300010 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.300395 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300495 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.300542 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.300752 (+ 210us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.300817 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:31:02.300940 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.297495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.243.131:51265 (local address 127.16.243.129:35339)
0618 22:31:02.297602 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.297606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.297724 (+ 118us) server_negotiation.cc:407] Connection header received
0618 22:31:02.297794 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.297797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.297850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.297937 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:31:02.298221 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.298858 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300028 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.300386 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300484 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.300542 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.300765 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.300830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:31:02.300964 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.297449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.130:36339 (local address 127.16.243.131:54417)
0618 22:31:02.297874 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.297887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.298004 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.298264 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.298266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.298283 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:02.298388 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.298394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.299103 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.299105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.299901 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.299910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.300498 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.300645 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.300664 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.300723 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.300792 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":41,"spinlock_wait_cycles":32384,"thread_start_us":124,"threads_started":1}
I20260618 22:31:02.300940 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.297386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35339 (local address 127.16.243.131:51265)
0618 22:31:02.297575 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.297589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:02.297714 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:02.297953 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:02.297955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:02.298001 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:31:02.298116 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.298124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.298951 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.298955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.299910 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:02.299920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.300490 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.300493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:02.300658 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.300675 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:31:02.300741 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:31:02.300807 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":63,"spinlock_wait_cycles":5120}
I20260618 22:31:02.301678 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5167835f43fd4328ba7f7966e5373376" candidate_uuid: "d856c8fbe84b404c8159de0bdf3681ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624e4decb7b74175a01be3be27dad992" is_pre_election: true
I20260618 22:31:02.301683 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5167835f43fd4328ba7f7966e5373376" candidate_uuid: "d856c8fbe84b404c8159de0bdf3681ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5173162f82364e3aa217fa63752a2266" is_pre_election: true
I20260618 22:31:02.301951 4903 raft_consensus.cc:2454] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d856c8fbe84b404c8159de0bdf3681ef in term 0.
I20260618 22:31:02.301957 4772 raft_consensus.cc:2454] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d856c8fbe84b404c8159de0bdf3681ef in term 0.
I20260618 22:31:02.302392 4969 leader_election.cc:302] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5173162f82364e3aa217fa63752a2266, d856c8fbe84b404c8159de0bdf3681ef; no voters:
I20260618 22:31:02.302668 5105 raft_consensus.cc:2781] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:31:02.302752 5105 raft_consensus.cc:482] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:31:02.302798 5105 raft_consensus.cc:3037] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:02.303784 5105 raft_consensus.cc:504] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.304085 5105 leader_election.cc:288] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [CANDIDATE]: Term 1 election: Requested vote from peers 624e4decb7b74175a01be3be27dad992 (127.16.243.129:35339), 5173162f82364e3aa217fa63752a2266 (127.16.243.130:36339)
I20260618 22:31:02.304380 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5167835f43fd4328ba7f7966e5373376" candidate_uuid: "d856c8fbe84b404c8159de0bdf3681ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "624e4decb7b74175a01be3be27dad992"
I20260618 22:31:02.304425 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5167835f43fd4328ba7f7966e5373376" candidate_uuid: "d856c8fbe84b404c8159de0bdf3681ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5173162f82364e3aa217fa63752a2266"
I20260618 22:31:02.304488 4772 raft_consensus.cc:3037] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:02.304522 4903 raft_consensus.cc:3037] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:02.305439 4772 raft_consensus.cc:2454] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d856c8fbe84b404c8159de0bdf3681ef in term 1.
I20260618 22:31:02.305442 4903 raft_consensus.cc:2454] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d856c8fbe84b404c8159de0bdf3681ef in term 1.
I20260618 22:31:02.305706 4970 leader_election.cc:302] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 624e4decb7b74175a01be3be27dad992, d856c8fbe84b404c8159de0bdf3681ef; no voters:
I20260618 22:31:02.305858 5105 raft_consensus.cc:2781] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:02.306058 5105 raft_consensus.cc:686] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 1 LEADER]: Becoming Leader. State: Replica: d856c8fbe84b404c8159de0bdf3681ef, State: Running, Role: LEADER
I20260618 22:31:02.306378 5105 consensus_queue.cc:237] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } }
I20260618 22:31:02.309437 4630 catalog_manager.cc:5495] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef reported cstate change: term changed from 0 to 1, leader changed from <none> to d856c8fbe84b404c8159de0bdf3681ef (127.16.243.131). New cstate: current_term: 1 leader_uuid: "d856c8fbe84b404c8159de0bdf3681ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d856c8fbe84b404c8159de0bdf3681ef" member_type: VOTER last_known_addr { host: "127.16.243.131" port: 36951 } health_report { overall_health: HEALTHY } } }
I20260618 22:31:02.339159 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.335975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.16.243.131:36951)
0618 22:31:02.336103 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:02.336108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:02.336123 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:31:02.336207 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:02.336210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:02.336279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:02.336380 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:02.336602 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.337092 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.337860 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:02.338053 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:02.338132 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:02.338388 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:02.338834 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:02.338918 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:02.338974 (+ 56us) server_negotiation.cc:299] Negotiation successful
0618 22:31:02.339031 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:31:02.344765 4772 raft_consensus.cc:1261] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 1 FOLLOWER]: Refusing update from remote peer d856c8fbe84b404c8159de0bdf3681ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:02.344765 4903 raft_consensus.cc:1261] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 1 FOLLOWER]: Refusing update from remote peer d856c8fbe84b404c8159de0bdf3681ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:02.345456 5105 consensus_queue.cc:1035] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "624e4decb7b74175a01be3be27dad992" member_type: VOTER last_known_addr { host: "127.16.243.129" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:02.345651 5107 consensus_queue.cc:1035] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5173162f82364e3aa217fa63752a2266" member_type: VOTER last_known_addr { host: "127.16.243.130" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:02.358651 5112 mvcc.cc:205] Tried to move back new op lower bound from 7298342348156846080 to 7298342348009541632. Current Snapshot: MvccSnapshot[applied={T|T < 7298342348156846080 or (T in {7298342348156846080})}]
I20260618 22:31:02.361213 4628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:31:02.361382 4628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:31:02.363682 4628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b63cce56511e4630922dcbdc5a8a4417: Sending DeleteTablet for 3 replicas of tablet 5167835f43fd4328ba7f7966e5373376
I20260618 22:31:02.364369 4752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5167835f43fd4328ba7f7966e5373376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:02 UTC) from {username='slave'} at 127.0.0.1:51748
I20260618 22:31:02.364369 4883 tablet_service.cc:1514] Processing DeleteTablet for tablet 5167835f43fd4328ba7f7966e5373376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:02 UTC) from {username='slave'} at 127.0.0.1:52458
I20260618 22:31:02.364550 5014 tablet_service.cc:1514] Processing DeleteTablet for tablet 5167835f43fd4328ba7f7966e5373376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:31:02 UTC) from {username='slave'} at 127.0.0.1:45060
I20260618 22:31:02.364888 5121 tablet_replica.cc:330] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992: stopping tablet replica
I20260618 22:31:02.364969 5122 tablet_replica.cc:330] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: stopping tablet replica
I20260618 22:31:02.365262 5122 raft_consensus.cc:2229] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:31:02.365262 5121 raft_consensus.cc:2229] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:31:02.365598 5122 raft_consensus.cc:2258] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:31:02.365713 5121 raft_consensus.cc:2258] T 5167835f43fd4328ba7f7966e5373376 P 624e4decb7b74175a01be3be27dad992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:31:02.365717 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4690
I20260618 22:31:02.366321 5120 tablet_replica.cc:330] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: stopping tablet replica
I20260618 22:31:02.366608 5120 raft_consensus.cc:2229] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:31:02.366882 5120 raft_consensus.cc:2258] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:31:02.367976 5122 ts_tablet_manager.cc:1905] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:31:02.368031 5120 ts_tablet_manager.cc:1905] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:31:02.370605 5120 ts_tablet_manager.cc:1918] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:31:02.370600 5122 ts_tablet_manager.cc:1918] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:31:02.370698 5120 log.cc:1196] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-1/wal/wals/5167835f43fd4328ba7f7966e5373376
I20260618 22:31:02.370801 5122 log.cc:1196] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Deleting WAL directory at /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821616719733-17358-0/minicluster-data/ts-2/wal/wals/5167835f43fd4328ba7f7966e5373376
I20260618 22:31:02.370981 5120 ts_tablet_manager.cc:1939] T 5167835f43fd4328ba7f7966e5373376 P 5173162f82364e3aa217fa63752a2266: Deleting consensus metadata
I20260618 22:31:02.371182 5122 ts_tablet_manager.cc:1939] T 5167835f43fd4328ba7f7966e5373376 P d856c8fbe84b404c8159de0bdf3681ef: Deleting consensus metadata
I20260618 22:31:02.374665 4614 catalog_manager.cc:4841] TS d856c8fbe84b404c8159de0bdf3681ef (127.16.243.131:36951): tablet 5167835f43fd4328ba7f7966e5373376 (table test-table [id=4d19a444d6ea4ab49d70566b3cf3c218]) successfully deleted
I20260618 22:31:02.374665 4616 catalog_manager.cc:4841] TS 5173162f82364e3aa217fa63752a2266 (127.16.243.130:36339): tablet 5167835f43fd4328ba7f7966e5373376 (table test-table [id=4d19a444d6ea4ab49d70566b3cf3c218]) successfully deleted
W20260618 22:31:02.375015 4617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.243.129:35339 (error 108)
I20260618 22:31:02.375844 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4821
I20260618 22:31:02.376019 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:02.375701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.129:35339 (local address 127.0.0.1:51750)
0618 22:31:02.375868 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:02.375938 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.129:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260618 22:31:02.377140 4617 catalog_manager.cc:4584] TS 624e4decb7b74175a01be3be27dad992 (127.16.243.129:35339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5167835f43fd4328ba7f7966e5373376: Network error: Client connection negotiation failed: client connection to 127.16.243.129:35339: connect: Connection refused (error 111)
I20260618 22:31:02.381191 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4952
I20260618 22:31:02.385913 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 4599
2026-06-18T22:31:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1024 ms)
[----------] 6 tests from EncryptionPolicyTest (9794 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:02 dist-test-slave-jm5g krb5kdc[5126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:02 dist-test-slave-jm5g krb5kdc[5126](info): set up 2 sockets
Jun 18 22:31:02 dist-test-slave-jm5g krb5kdc[5126](info): commencing operation
krb5kdc: starting...
W20260618 22:31:03.330263 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:03.440857 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:03.551644 5139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:03.551924 5139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:03.551982 5139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:03.556113 5139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:03.556182 5139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:03.556208 5139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:03.556229 5139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:03.556258 5139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:03.561563 5139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:03.562666 5139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:03.563773 5139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:03.572233 5145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:03.572227 5147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:03.572386 5139 server_base.cc:1034] running on GCE node
W20260618 22:31:03.572227 5144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:03.573097 5139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:03.574193 5139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:03.575354 5139 hybrid_clock.cc:648] HybridClock initialized: now 1781821863575337 us; error 29 us; skew 500 ppm
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:03.578058 5139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:03.579008 5139 webserver.cc:460] Webserver started at http://127.0.0.1:44987/ using document root <none> and password file <none>
I20260618 22:31:03.579545 5139 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:03.579615 5139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:03.579802 5139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:03.581810 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "790df4028f38476195d41bb9f9ed267a"
format_stamp: "Formatted at 2026-06-18 22:31:03 on dist-test-slave-jm5g"
server_key: "1d5d1f9b7735f44e2e7ec5ce84bc0912"
server_key_iv: "72aa2dac3a84b1fe0ef474fc06513a54"
server_key_version: "encryptionkey@0"
I20260618 22:31:03.582332 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "790df4028f38476195d41bb9f9ed267a"
format_stamp: "Formatted at 2026-06-18 22:31:03 on dist-test-slave-jm5g"
server_key: "1d5d1f9b7735f44e2e7ec5ce84bc0912"
server_key_iv: "72aa2dac3a84b1fe0ef474fc06513a54"
server_key_version: "encryptionkey@0"
I20260618 22:31:03.585928 5139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:31:03.588346 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:03.589576 5139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:03.589687 5139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "790df4028f38476195d41bb9f9ed267a"
format_stamp: "Formatted at 2026-06-18 22:31:03 on dist-test-slave-jm5g"
server_key: "1d5d1f9b7735f44e2e7ec5ce84bc0912"
server_key_iv: "72aa2dac3a84b1fe0ef474fc06513a54"
server_key_version: "encryptionkey@0"
I20260618 22:31:03.589778 5139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:03.615651 5139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:03.618727 5139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:03.618906 5139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:03.625743 5139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43471
I20260618 22:31:03.625761 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43471 every 8 connection(s)
I20260618 22:31:03.626736 5139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:03.629419 5207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:03.634795 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: Bootstrap starting.
I20260618 22:31:03.636966 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5139
I20260618 22:31:03.637095 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:03.637157 5207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:03.637341 17358 external_mini_cluster.cc:1428] Setting key 377735b15d1fde640454efe4ae962338
I20260618 22:31:03.637984 5207 log.cc:824] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:03.639899 5207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: No bootstrap required, opened a new log
I20260618 22:31:03.644039 5207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260618 22:31:03.644330 5207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:03.644392 5207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790df4028f38476195d41bb9f9ed267a, State: Initialized, Role: FOLLOWER
I20260618 22:31:03.644897 5207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260618 22:31:03.645071 5207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:03.645138 5207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:03.645241 5207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 0 FOLLOWER]: Advancing to term 1
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:31:03.646242 5207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260618 22:31:03.646657 5207 leader_election.cc:302] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 790df4028f38476195d41bb9f9ed267a; no voters:
I20260618 22:31:03.647006 5207 leader_election.cc:288] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:03.647097 5212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:03.647293 5212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [term 1 LEADER]: Becoming Leader. State: Replica: 790df4028f38476195d41bb9f9ed267a, State: Running, Role: LEADER
I20260618 22:31:03.647629 5212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260618 22:31:03.648275 5207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:03.649312 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 790df4028f38476195d41bb9f9ed267a. Latest consensus state: current_term: 1 leader_uuid: "790df4028f38476195d41bb9f9ed267a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } } }
I20260618 22:31:03.649449 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:03.649466 5213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "790df4028f38476195d41bb9f9ed267a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790df4028f38476195d41bb9f9ed267a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } } }
I20260618 22:31:03.649585 5213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:03.652316 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:03.638461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60323 (local address 127.0.0.1:43471)
0618 22:31:03.638897 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:03.638912 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:03.638946 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:31:03.639598 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:03.639614 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:03.639943 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:03.640251 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:03.641101 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:03.642032 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:03.643275 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:03.643762 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:03.643876 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:03.646689 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:03.646694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:03.646707 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:03.646735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:03.648796 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:03.649430 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:03.649437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:03.649444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:03.649738 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:03.650117 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:03.650121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:03.650123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:03.650665 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:03.650875 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:03.651183 (+ 308us) server_negotiation.cc:299] Negotiation successful
0618 22:31:03.651397 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
W20260618 22:31:03.653033 5227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:03.653100 5227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:03.653151 5221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:03.653899 5221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:03.655762 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:03.659250 5221 catalog_manager.cc:1347] Generated new cluster ID: 26fb5dfea0ad40e2a94977a95526a808
I20260618 22:31:03.659312 5221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:03.666253 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:03.657971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58871 (local address 127.0.0.1:43471)
0618 22:31:03.658080 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:03.658084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:03.658290 (+ 206us) server_negotiation.cc:407] Connection header received
0618 22:31:03.658390 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:03.658393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:03.658435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:03.658500 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:03.659211 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:03.659942 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:03.661038 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:03.661409 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:03.661517 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:03.663040 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:03.663044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:03.663047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:03.663073 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:03.664623 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:03.665086 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:03.665089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:03.665091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:03.665172 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:03.665443 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:03.665446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:03.665448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:03.665650 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:03.665793 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:03.666008 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 22:31:03.666098 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:31:03.670810 5221 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:03.672022 5221 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:03.677006 5221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 790df4028f38476195d41bb9f9ed267a: Generated new TSK 0
I20260618 22:31:03.677713 5221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:03.685505 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5139
W20260618 22:31:03.689759 5124 connection.cc:535] client connection to 127.0.0.1:43471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43471): Connection reset by peer (error 104)
2026-06-18T22:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5238](info): set up 2 sockets
Jun 18 22:31:03 dist-test-slave-jm5g krb5kdc[5238](info): commencing operation
krb5kdc: starting...
W20260618 22:31:04.638515 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:04 dist-test-slave-jm5g krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:04.751396 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:04.861450 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:04.861739 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:04.861783 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:04.865811 5251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:04.865880 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:04.865898 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:04.865914 5251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:04.865935 5251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:04.871562 5251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:04.872572 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:04.873522 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:04.881681 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:04.881685 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:04.881681 5256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:04.881960 5251 server_base.cc:1034] running on GCE node
I20260618 22:31:04.882535 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:04.883730 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:04.884928 5251 hybrid_clock.cc:648] HybridClock initialized: now 1781821864884908 us; error 31 us; skew 500 ppm
Jun 18 22:31:04 dist-test-slave-jm5g krb5kdc[5238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:04.897580 5251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:04.898594 5251 webserver.cc:460] Webserver started at http://127.0.0.1:34525/ using document root <none> and password file <none>
I20260618 22:31:04.899152 5251 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:04.899214 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:04.899385 5251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:04.901731 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f"
format_stamp: "Formatted at 2026-06-18 22:31:04 on dist-test-slave-jm5g"
server_key: "ce5b0e30c0b88197b3e0d02b5ab1a375"
server_key_iv: "c6b80f4a7a79904b27029000edde3882"
server_key_version: "encryptionkey@0"
I20260618 22:31:04.902251 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f"
format_stamp: "Formatted at 2026-06-18 22:31:04 on dist-test-slave-jm5g"
server_key: "ce5b0e30c0b88197b3e0d02b5ab1a375"
server_key_iv: "c6b80f4a7a79904b27029000edde3882"
server_key_version: "encryptionkey@0"
I20260618 22:31:04.905993 5251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:31:04.908510 5266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:04.909762 5251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:04.909873 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f"
format_stamp: "Formatted at 2026-06-18 22:31:04 on dist-test-slave-jm5g"
server_key: "ce5b0e30c0b88197b3e0d02b5ab1a375"
server_key_iv: "c6b80f4a7a79904b27029000edde3882"
server_key_version: "encryptionkey@0"
I20260618 22:31:04.909965 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:04.940301 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:04.943569 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:04.943729 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:04.950448 5251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40495
I20260618 22:31:04.950460 5318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40495 every 8 connection(s)
I20260618 22:31:04.951440 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:04.954075 5319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:04.957592 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5251
I20260618 22:31:04.957711 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:04.957942 17358 external_mini_cluster.cc:1428] Setting key e471241aea92abbd99cafa01709b895f
I20260618 22:31:04.959645 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f: Bootstrap starting.
I20260618 22:31:04.961894 5319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:04.962672 5319 log.cc:824] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:04.964730 5319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f: No bootstrap required, opened a new log
I20260618 22:31:04.966809 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:04.959288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52685 (local address 127.0.0.1:40495)
0618 22:31:04.959677 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:04.959693 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:04.959726 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:31:04.960366 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:04.960382 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:04.960723 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:04.961034 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:04.961412 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:04.962411 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:04.963466 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:04.963937 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:04.964061 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:04.964936 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:04.964940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:04.964953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:04.965336 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:04.965387 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:04.965408 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:04.965525 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:04.965681 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:31:04.965813 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260618 22:31:04.967851 5319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260618 22:31:04.968120 5319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:04.968175 5319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa6b31631dd42c3908afc1bf7a1cd0f, State: Initialized, Role: FOLLOWER
I20260618 22:31:04.968614 5319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260618 22:31:04.968763 5319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:04.968876 5319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:04.968977 5319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:04.969729 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:04.970062 5319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260618 22:31:04.970376 5319 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa6b31631dd42c3908afc1bf7a1cd0f; no voters:
I20260618 22:31:04.970687 5319 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:04.970762 5324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:04.970932 5324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [term 1 LEADER]: Becoming Leader. State: Replica: 5aa6b31631dd42c3908afc1bf7a1cd0f, State: Running, Role: LEADER
I20260618 22:31:04.971302 5324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } }
I20260618 22:31:04.971953 5319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:04.972913 5327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa6b31631dd42c3908afc1bf7a1cd0f. Latest consensus state: current_term: 1 leader_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } } }
I20260618 22:31:04.973043 5327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:04.973376 5328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa6b31631dd42c3908afc1bf7a1cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40495 } } }
I20260618 22:31:04.973460 5328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:04.973779 5338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:04.976161 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:04.971746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51503 (local address 127.0.0.1:40495)
0618 22:31:04.971852 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:04.971856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:04.972055 (+ 199us) server_negotiation.cc:407] Connection header received
0618 22:31:04.972208 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:04.972211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:04.972255 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:04.972317 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:04.972659 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:04.973433 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:04.974534 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:04.974826 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:04.974903 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:04.975719 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:04.975723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:04.975725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:04.975738 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:04.975765 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:04.975777 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:04.975864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:04.975998 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:31:04.976033 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:31:04.977699 5338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:04.982668 5338 catalog_manager.cc:1347] Generated new cluster ID: 0633ad6f4e724e9f90c480b367baeceb
I20260618 22:31:04.982789 5338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:04.993044 5338 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:04.993945 5338 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:05.002635 5338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa6b31631dd42c3908afc1bf7a1cd0f: Generated new TSK 0
I20260618 22:31:05.003362 5338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:05.014506 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5251
W20260618 22:31:05.019527 5236 connection.cc:535] client connection to 127.0.0.1:40495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40495): Connection reset by peer (error 104)
2026-06-18T22:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:05 dist-test-slave-jm5g krb5kdc[5350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:05 dist-test-slave-jm5g krb5kdc[5350](info): set up 2 sockets
Jun 18 22:31:05 dist-test-slave-jm5g krb5kdc[5350](info): commencing operation
krb5kdc: starting...
W20260618 22:31:05.978367 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:06 dist-test-slave-jm5g krb5kdc[5350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:06.089387 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:06.198380 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:06.198683 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:06.198742 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:06.202713 5363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:06.202782 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:06.202809 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:06.202831 5363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:06.202852 5363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:06.208235 5363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:06.209321 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:06.210338 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:06.218665 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:06.218660 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:06.218725 5363 server_base.cc:1034] running on GCE node
W20260618 22:31:06.218660 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:06.219424 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:06.220481 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:06.221642 5363 hybrid_clock.cc:648] HybridClock initialized: now 1781821866221626 us; error 32 us; skew 500 ppm
Jun 18 22:31:06 dist-test-slave-jm5g krb5kdc[5350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:06.224418 5363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:06.225301 5363 webserver.cc:460] Webserver started at http://127.0.0.1:39615/ using document root <none> and password file <none>
I20260618 22:31:06.225835 5363 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:06.225900 5363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:06.226107 5363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:06.228166 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "b0384f6c73ef4fe5a21af8527c886441"
format_stamp: "Formatted at 2026-06-18 22:31:06 on dist-test-slave-jm5g"
server_key: "852f42c32f2ca4154c57560c7846e6a1"
server_key_iv: "f0d0a2f522e7bbe5005e6abc9babe32f"
server_key_version: "encryptionkey@0"
I20260618 22:31:06.228694 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "b0384f6c73ef4fe5a21af8527c886441"
format_stamp: "Formatted at 2026-06-18 22:31:06 on dist-test-slave-jm5g"
server_key: "852f42c32f2ca4154c57560c7846e6a1"
server_key_iv: "f0d0a2f522e7bbe5005e6abc9babe32f"
server_key_version: "encryptionkey@0"
I20260618 22:31:06.232440 5363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:31:06.235105 5378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:06.236438 5363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:06.236559 5363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "b0384f6c73ef4fe5a21af8527c886441"
format_stamp: "Formatted at 2026-06-18 22:31:06 on dist-test-slave-jm5g"
server_key: "852f42c32f2ca4154c57560c7846e6a1"
server_key_iv: "f0d0a2f522e7bbe5005e6abc9babe32f"
server_key_version: "encryptionkey@0"
I20260618 22:31:06.236667 5363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:06.241328 5363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:06.244451 5363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:06.244627 5363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:06.251197 5363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46315
I20260618 22:31:06.251210 5430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46315 every 8 connection(s)
I20260618 22:31:06.252130 5363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:06.254944 5431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:06.254944 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5363
I20260618 22:31:06.255081 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:06.255362 17358 external_mini_cluster.cc:1428] Setting key af0568e905068e3f667d7c26526ccc8b
I20260618 22:31:06.260551 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:06.256625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36111 (local address 127.0.0.1:46315)
0618 22:31:06.257037 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:06.257055 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:06.257097 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:31:06.257791 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:06.257816 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:06.258191 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:06.258589 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:06.259223 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:06.259228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:06.259242 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:06.259252 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:06.259304 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:06.259328 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:06.259458 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:06.259526 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 22:31:06.259671 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260618 22:31:06.262070 5431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441: Bootstrap starting.
I20260618 22:31:06.263235 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:06.264601 5431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:06.265413 5431 log.cc:824] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:06.266897 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:06.265357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51581 (local address 127.0.0.1:46315)
0618 22:31:06.265472 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:06.265477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:06.265703 (+ 226us) server_negotiation.cc:407] Connection header received
0618 22:31:06.265822 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:06.265825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:06.265863 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:06.265939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:06.266528 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:06.266531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:06.266533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:06.266543 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:06.266606 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:06.266621 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:06.266686 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:06.266763 (+ 77us) server_negotiation.cc:299] Negotiation successful
0618 22:31:06.266792 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:31:06.267642 5431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441: No bootstrap required, opened a new log
I20260618 22:31:06.271371 5431 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260618 22:31:06.271601 5431 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:06.271642 5431 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0384f6c73ef4fe5a21af8527c886441, State: Initialized, Role: FOLLOWER
I20260618 22:31:06.271993 5431 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260618 22:31:06.272099 5431 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:06.272142 5431 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:06.272208 5431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:06.273023 5431 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260618 22:31:06.273314 5431 leader_election.cc:302] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0384f6c73ef4fe5a21af8527c886441; no voters:
I20260618 22:31:06.273602 5431 leader_election.cc:288] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:06.273692 5441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:06.273890 5441 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [term 1 LEADER]: Becoming Leader. State: Replica: b0384f6c73ef4fe5a21af8527c886441, State: Running, Role: LEADER
I20260618 22:31:06.274197 5441 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260618 22:31:06.274760 5431 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:06.275782 5443 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0384f6c73ef4fe5a21af8527c886441. Latest consensus state: current_term: 1 leader_uuid: "b0384f6c73ef4fe5a21af8527c886441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260618 22:31:06.275802 5442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0384f6c73ef4fe5a21af8527c886441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0384f6c73ef4fe5a21af8527c886441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260618 22:31:06.275964 5443 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:06.275965 5442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:06.276347 5450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:06.279088 5450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:06.284514 5450 catalog_manager.cc:1347] Generated new cluster ID: 129c6785cebb4ba5a573990d03048867
I20260618 22:31:06.284585 5450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:06.298610 5450 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:06.298918 5450 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:06.303232 5450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0384f6c73ef4fe5a21af8527c886441: Generated new TSK 0
I20260618 22:31:06.303866 5450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:06.308027 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5363
2026-06-18T22:31:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:06 dist-test-slave-jm5g krb5kdc[5462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:06 dist-test-slave-jm5g krb5kdc[5462](info): set up 2 sockets
Jun 18 22:31:06 dist-test-slave-jm5g krb5kdc[5462](info): commencing operation
krb5kdc: starting...
W20260618 22:31:07.250471 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:07.363590 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:07.471269 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:07.471531 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:07.471572 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:07.475555 5475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:07.475613 5475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:07.475631 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:07.475646 5475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:07.475662 5475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:07.480881 5475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:07.481889 5475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:07.482905 5475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:07.491184 5480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:07.491221 5475 server_base.cc:1034] running on GCE node
W20260618 22:31:07.491166 5481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:07.491166 5483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:07.491981 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:07.493093 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:07.494247 5475 hybrid_clock.cc:648] HybridClock initialized: now 1781821867494231 us; error 32 us; skew 500 ppm
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:07.496886 5475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:07.497737 5475 webserver.cc:460] Webserver started at http://127.0.0.1:35199/ using document root <none> and password file <none>
I20260618 22:31:07.498260 5475 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:07.498322 5475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:07.498502 5475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:07.500478 5475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "7729f7ec10bc44cbb94bb057f280df36"
format_stamp: "Formatted at 2026-06-18 22:31:07 on dist-test-slave-jm5g"
server_key: "6b53e8f1aec5cb21f62845f2067ff0d2"
server_key_iv: "e856dd17aebf75daa994dc66f145a4ab"
server_key_version: "encryptionkey@0"
I20260618 22:31:07.501003 5475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "7729f7ec10bc44cbb94bb057f280df36"
format_stamp: "Formatted at 2026-06-18 22:31:07 on dist-test-slave-jm5g"
server_key: "6b53e8f1aec5cb21f62845f2067ff0d2"
server_key_iv: "e856dd17aebf75daa994dc66f145a4ab"
server_key_version: "encryptionkey@0"
I20260618 22:31:07.504585 5475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:31:07.506892 5490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:07.508105 5475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:31:07.508213 5475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "7729f7ec10bc44cbb94bb057f280df36"
format_stamp: "Formatted at 2026-06-18 22:31:07 on dist-test-slave-jm5g"
server_key: "6b53e8f1aec5cb21f62845f2067ff0d2"
server_key_iv: "e856dd17aebf75daa994dc66f145a4ab"
server_key_version: "encryptionkey@0"
I20260618 22:31:07.508303 5475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:07.520949 5475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:07.524027 5475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:07.524199 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:07.530725 5475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44833
I20260618 22:31:07.530778 5542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44833 every 8 connection(s)
I20260618 22:31:07.531656 5475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:07.534569 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:07.539350 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5475
I20260618 22:31:07.539459 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:07.539690 17358 external_mini_cluster.cc:1428] Setting key 4179c2db84efe10bdc026fd82c55daf8
I20260618 22:31:07.540095 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36: Bootstrap starting.
I20260618 22:31:07.542488 5543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:07.543227 5543 log.cc:824] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:07.545024 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36: No bootstrap required, opened a new log
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:31:07.548759 5543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260618 22:31:07.548980 5543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:07.549026 5543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7729f7ec10bc44cbb94bb057f280df36, State: Initialized, Role: FOLLOWER
I20260618 22:31:07.549443 5543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260618 22:31:07.549569 5543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:07.549669 5543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:07.549782 5543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:07.550877 5543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260618 22:31:07.551230 5543 leader_election.cc:302] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7729f7ec10bc44cbb94bb057f280df36; no voters:
I20260618 22:31:07.551555 5543 leader_election.cc:288] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:07.551627 5548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:07.551815 5548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [term 1 LEADER]: Becoming Leader. State: Replica: 7729f7ec10bc44cbb94bb057f280df36, State: Running, Role: LEADER
I20260618 22:31:07.552179 5548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260618 22:31:07.552881 5543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:07.553850 5550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7729f7ec10bc44cbb94bb057f280df36. Latest consensus state: current_term: 1 leader_uuid: "7729f7ec10bc44cbb94bb057f280df36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260618 22:31:07.553966 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:07.541008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37591 (local address 127.0.0.1:44833)
0618 22:31:07.541448 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:07.541463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:07.541496 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:31:07.542117 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:07.542135 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:07.542469 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:07.542815 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:07.543586 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:07.544582 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:07.545429 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:07.545883 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:07.546022 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:07.548616 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:07.548622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:07.548635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:07.548662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:07.550611 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:07.551011 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:07.551016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:07.551023 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:07.551163 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:07.551424 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:07.551427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:07.551428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:07.551876 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:07.552077 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:07.552737 (+ 660us) server_negotiation.cc:299] Negotiation successful
0618 22:31:07.553024 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260618 22:31:07.553970 5550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:07.554041 5549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7729f7ec10bc44cbb94bb057f280df36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7729f7ec10bc44cbb94bb057f280df36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260618 22:31:07.554101 5549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:07.554388 5557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:07.557066 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:07.557659 5557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:07.563971 5557 catalog_manager.cc:1347] Generated new cluster ID: 1c0af5df1a44405f868049ca1542d96d
I20260618 22:31:07.564049 5557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:07.568625 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:07.560311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33491 (local address 127.0.0.1:44833)
0618 22:31:07.560424 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:07.560428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:07.560827 (+ 399us) server_negotiation.cc:407] Connection header received
0618 22:31:07.561028 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:07.561032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:07.561084 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:07.561174 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:07.561921 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:07.562590 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:07.563374 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:07.563632 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:07.563722 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:07.565378 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:07.565382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:07.565385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:07.565412 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:07.567049 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:07.567482 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:07.567485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:07.567487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:07.567573 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:07.567806 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:07.567809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:07.567810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:07.568032 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:07.568146 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:07.568352 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 22:31:07.568469 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:31:07.581334 5557 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:07.582299 5557 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:07.593133 5557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7729f7ec10bc44cbb94bb057f280df36: Generated new TSK 0
I20260618 22:31:07.593860 5557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:07.598459 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5475
2026-06-18T22:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5574](info): set up 2 sockets
Jun 18 22:31:07 dist-test-slave-jm5g krb5kdc[5574](info): commencing operation
krb5kdc: starting...
W20260618 22:31:08.546437 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:08 dist-test-slave-jm5g krb5kdc[5574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:08.655647 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:08.762880 5587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:08.763221 5587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:08.763281 5587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:08.767450 5587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:08.767518 5587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:08.767537 5587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:08.767560 5587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:08.767575 5587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:08.773010 5587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:08.774049 5587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:08.775029 5587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:08.783001 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:08.783139 5587 server_base.cc:1034] running on GCE node
W20260618 22:31:08.783011 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:08.782996 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:08.783699 5587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:08.784833 5587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:08.785995 5587 hybrid_clock.cc:648] HybridClock initialized: now 1781821868785983 us; error 26 us; skew 500 ppm
Jun 18 22:31:08 dist-test-slave-jm5g krb5kdc[5574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:08.788734 5587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:08.789609 5587 webserver.cc:460] Webserver started at http://127.0.0.1:41287/ using document root <none> and password file <none>
I20260618 22:31:08.790120 5587 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:08.790177 5587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:08.790338 5587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:08.792413 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "167c6a906ca844649a7b39d192494f99"
format_stamp: "Formatted at 2026-06-18 22:31:08 on dist-test-slave-jm5g"
server_key: "53f54fe25cb683bd83fde81cca7d01b4"
server_key_iv: "1b47456e7612f6ffed758c48f551e6b7"
server_key_version: "encryptionkey@0"
I20260618 22:31:08.792904 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "167c6a906ca844649a7b39d192494f99"
format_stamp: "Formatted at 2026-06-18 22:31:08 on dist-test-slave-jm5g"
server_key: "53f54fe25cb683bd83fde81cca7d01b4"
server_key_iv: "1b47456e7612f6ffed758c48f551e6b7"
server_key_version: "encryptionkey@0"
I20260618 22:31:08.796425 5587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:31:08.798652 5602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:08.799803 5587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:08.799913 5587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "167c6a906ca844649a7b39d192494f99"
format_stamp: "Formatted at 2026-06-18 22:31:08 on dist-test-slave-jm5g"
server_key: "53f54fe25cb683bd83fde81cca7d01b4"
server_key_iv: "1b47456e7612f6ffed758c48f551e6b7"
server_key_version: "encryptionkey@0"
I20260618 22:31:08.800009 5587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:08.816169 5587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:08.818945 5587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:08.819121 5587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:08.825834 5587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37933
I20260618 22:31:08.825845 5654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37933 every 8 connection(s)
I20260618 22:31:08.826788 5587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:08.829402 5655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:08.831038 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5587
I20260618 22:31:08.831153 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:08.831374 17358 external_mini_cluster.cc:1428] Setting key 79df65c8769ca997a9d7c236e0572b9e
I20260618 22:31:08.835317 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99: Bootstrap starting.
I20260618 22:31:08.837422 5655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:08.838203 5655 log.cc:824] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:08.839855 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:08.832711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37771 (local address 127.0.0.1:37933)
0618 22:31:08.833143 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:08.833158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:08.833193 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:08.833830 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:08.833848 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:08.834188 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:08.834489 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:08.834921 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:08.835899 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:08.836750 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:08.837209 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:08.837360 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:08.838171 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:08.838176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:08.838189 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:08.838630 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:08.838687 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:08.838710 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:08.838795 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:08.838925 (+ 130us) server_negotiation.cc:299] Negotiation successful
0618 22:31:08.839128 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260618 22:31:08.840107 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99: No bootstrap required, opened a new log
I20260618 22:31:08.842396 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:08.843657 5655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } }
I20260618 22:31:08.843892 5655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:08.843955 5655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167c6a906ca844649a7b39d192494f99, State: Initialized, Role: FOLLOWER
I20260618 22:31:08.844545 5655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } }
I20260618 22:31:08.844683 5655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:08.844743 5655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:08.844834 5655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:08.845890 5655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } }
I20260618 22:31:08.846263 5655 leader_election.cc:302] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 167c6a906ca844649a7b39d192494f99; no voters:
I20260618 22:31:08.846599 5655 leader_election.cc:288] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:08.846657 5665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:08.846841 5665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [term 1 LEADER]: Becoming Leader. State: Replica: 167c6a906ca844649a7b39d192494f99, State: Running, Role: LEADER
I20260618 22:31:08.847141 5665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } }
I20260618 22:31:08.847831 5655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:08.848264 5665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 167c6a906ca844649a7b39d192494f99. Latest consensus state: current_term: 1 leader_uuid: "167c6a906ca844649a7b39d192494f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } } }
I20260618 22:31:08.848369 5665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:08.848541 5665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "167c6a906ca844649a7b39d192494f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "167c6a906ca844649a7b39d192494f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37933 } } }
I20260618 22:31:08.848620 5665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:08.850140 5669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:08.851226 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:08.844243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50967 (local address 127.0.0.1:37933)
0618 22:31:08.844353 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:08.844358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:08.846816 (+ 2458us) server_negotiation.cc:407] Connection header received
0618 22:31:08.846993 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:08.846997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:08.847044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:08.847092 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:08.847488 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:08.848126 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:08.849210 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:08.849531 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:08.849630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:08.850511 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:08.850514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:08.850516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:08.850527 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:08.850576 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:08.850590 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:08.850654 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:08.851035 (+ 381us) server_negotiation.cc:299] Negotiation successful
0618 22:31:08.851102 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 22:31:08.853644 5669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:08.858981 5669 catalog_manager.cc:1347] Generated new cluster ID: ab18c87666294a71bfa5dd0adf47d028
I20260618 22:31:08.859043 5669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:08.878242 5669 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:08.879226 5669 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:08.890800 5669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 167c6a906ca844649a7b39d192494f99: Generated new TSK 0
I20260618 22:31:08.891705 5669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:08.902642 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5587
2026-06-18T22:31:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:08 dist-test-slave-jm5g krb5kdc[5686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:08 dist-test-slave-jm5g krb5kdc[5686](info): set up 2 sockets
Jun 18 22:31:08 dist-test-slave-jm5g krb5kdc[5686](info): commencing operation
krb5kdc: starting...
W20260618 22:31:09.854593 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:09 dist-test-slave-jm5g krb5kdc[5686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:09.966080 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:10.072201 5699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:10.072479 5699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:10.072538 5699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:10.076637 5699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:10.076705 5699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:10.076731 5699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:10.076753 5699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:10.076782 5699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:10.082140 5699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:10.083230 5699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:10.084208 5699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:10.092201 5705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:10.092216 5704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:10.092209 5707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:10.092386 5699 server_base.cc:1034] running on GCE node
I20260618 22:31:10.093021 5699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:10.094056 5699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:10.095214 5699 hybrid_clock.cc:648] HybridClock initialized: now 1781821870095198 us; error 28 us; skew 500 ppm
Jun 18 22:31:10 dist-test-slave-jm5g krb5kdc[5686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:10.102829 5699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:10.103771 5699 webserver.cc:460] Webserver started at http://127.0.0.1:39241/ using document root <none> and password file <none>
I20260618 22:31:10.104305 5699 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:10.104364 5699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:10.104542 5699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:10.106580 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "ff02125a1ffa4a9ea2a73218669530ef"
format_stamp: "Formatted at 2026-06-18 22:31:10 on dist-test-slave-jm5g"
server_key: "38667bcfead35a9a213f9b0cee4911f6"
server_key_iv: "1f2accc609b906b12d625b607b18d741"
server_key_version: "encryptionkey@0"
I20260618 22:31:10.107123 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "ff02125a1ffa4a9ea2a73218669530ef"
format_stamp: "Formatted at 2026-06-18 22:31:10 on dist-test-slave-jm5g"
server_key: "38667bcfead35a9a213f9b0cee4911f6"
server_key_iv: "1f2accc609b906b12d625b607b18d741"
server_key_version: "encryptionkey@0"
I20260618 22:31:10.110682 5699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:31:10.113004 5714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:10.114375 5699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:10.114502 5699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "ff02125a1ffa4a9ea2a73218669530ef"
format_stamp: "Formatted at 2026-06-18 22:31:10 on dist-test-slave-jm5g"
server_key: "38667bcfead35a9a213f9b0cee4911f6"
server_key_iv: "1f2accc609b906b12d625b607b18d741"
server_key_version: "encryptionkey@0"
I20260618 22:31:10.114658 5699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:10.119186 5699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:10.121915 5699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:10.122082 5699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:10.128744 5699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39653
I20260618 22:31:10.128757 5766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39653 every 8 connection(s)
I20260618 22:31:10.129683 5699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:10.131598 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5699
I20260618 22:31:10.131734 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:10.132006 17358 external_mini_cluster.cc:1428] Setting key 124c51e5c0f970b00b15b126c4633bdc
I20260618 22:31:10.132754 5767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:10.137188 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:10.133388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38451 (local address 127.0.0.1:39653)
0618 22:31:10.133846 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:10.133861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:10.133896 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:10.134625 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:10.134646 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:10.134932 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:10.135210 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:10.136026 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:10.136031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:10.136043 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:10.136053 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:10.136109 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:10.136129 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:10.136236 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:10.136336 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 22:31:10.136479 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260618 22:31:10.139058 5767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef: Bootstrap starting.
I20260618 22:31:10.139914 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:10.141713 5767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:10.142608 5767 log.cc:824] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:10.143693 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:10.142117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41195 (local address 127.0.0.1:39653)
0618 22:31:10.142239 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:10.142243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:10.142419 (+ 176us) server_negotiation.cc:407] Connection header received
0618 22:31:10.142578 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:10.142581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:10.142618 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:10.142680 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:10.143342 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:10.143346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:10.143349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:10.143360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:10.143386 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:10.143398 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:10.143484 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:10.143519 (+ 35us) server_negotiation.cc:299] Negotiation successful
0618 22:31:10.143556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:31:10.144989 5767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef: No bootstrap required, opened a new log
I20260618 22:31:10.148592 5767 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260618 22:31:10.148844 5767 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:10.148891 5767 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff02125a1ffa4a9ea2a73218669530ef, State: Initialized, Role: FOLLOWER
I20260618 22:31:10.149292 5767 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260618 22:31:10.149406 5767 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:10.149449 5767 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:10.149518 5767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:10.150589 5767 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260618 22:31:10.150967 5767 leader_election.cc:302] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff02125a1ffa4a9ea2a73218669530ef; no voters:
I20260618 22:31:10.151294 5767 leader_election.cc:288] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:10.151393 5777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:10.151619 5777 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [term 1 LEADER]: Becoming Leader. State: Replica: ff02125a1ffa4a9ea2a73218669530ef, State: Running, Role: LEADER
I20260618 22:31:10.151952 5777 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260618 22:31:10.152490 5767 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:10.153458 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff02125a1ffa4a9ea2a73218669530ef. Latest consensus state: current_term: 1 leader_uuid: "ff02125a1ffa4a9ea2a73218669530ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } } }
I20260618 22:31:10.153595 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:10.153903 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff02125a1ffa4a9ea2a73218669530ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff02125a1ffa4a9ea2a73218669530ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } } }
I20260618 22:31:10.154034 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:10.153950 5786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:10.156953 5786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:10.163791 5786 catalog_manager.cc:1347] Generated new cluster ID: fc34ba57f081492c8f6d278a8e3171c2
I20260618 22:31:10.163884 5786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:10.169948 5786 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:10.170383 5786 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:10.181501 5786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff02125a1ffa4a9ea2a73218669530ef: Generated new TSK 0
I20260618 22:31:10.182241 5786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:10.189733 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5699
2026-06-18T22:31:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:10 dist-test-slave-jm5g krb5kdc[5798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:10 dist-test-slave-jm5g krb5kdc[5798](info): set up 2 sockets
Jun 18 22:31:10 dist-test-slave-jm5g krb5kdc[5798](info): commencing operation
krb5kdc: starting...
W20260618 22:31:11.134495 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:11.248993 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:11.358341 5811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:11.358719 5811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:11.358785 5811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:11.362893 5811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:11.362973 5811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:11.362999 5811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:11.363019 5811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:11.363039 5811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:11.368695 5811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:11.369840 5811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:11.370898 5811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:11.379536 5816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:11.379593 5811 server_base.cc:1034] running on GCE node
W20260618 22:31:11.379550 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:11.379601 5819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:11.380393 5811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:11.381654 5811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:11.382812 5811 hybrid_clock.cc:648] HybridClock initialized: now 1781821871382785 us; error 44 us; skew 500 ppm
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:11.385574 5811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:11.386504 5811 webserver.cc:460] Webserver started at http://127.0.0.1:39143/ using document root <none> and password file <none>
I20260618 22:31:11.387095 5811 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:11.387157 5811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:11.387349 5811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:11.389423 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "807580e729b84f4c8ec9c540a1d5e12a"
format_stamp: "Formatted at 2026-06-18 22:31:11 on dist-test-slave-jm5g"
server_key: "4e32fce481497f6f52433f45bd50b8df"
server_key_iv: "b05951c867f45c5664e0f3ee8f183246"
server_key_version: "encryptionkey@0"
I20260618 22:31:11.389977 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "807580e729b84f4c8ec9c540a1d5e12a"
format_stamp: "Formatted at 2026-06-18 22:31:11 on dist-test-slave-jm5g"
server_key: "4e32fce481497f6f52433f45bd50b8df"
server_key_iv: "b05951c867f45c5664e0f3ee8f183246"
server_key_version: "encryptionkey@0"
I20260618 22:31:11.393757 5811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260618 22:31:11.396301 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:11.397542 5811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:31:11.397663 5811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "807580e729b84f4c8ec9c540a1d5e12a"
format_stamp: "Formatted at 2026-06-18 22:31:11 on dist-test-slave-jm5g"
server_key: "4e32fce481497f6f52433f45bd50b8df"
server_key_iv: "b05951c867f45c5664e0f3ee8f183246"
server_key_version: "encryptionkey@0"
I20260618 22:31:11.397773 5811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:11.410517 5811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:11.413475 5811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:11.413661 5811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:11.420430 5811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46331
I20260618 22:31:11.420442 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46331 every 8 connection(s)
I20260618 22:31:11.421396 5811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:11.424247 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:11.425275 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5811
I20260618 22:31:11.425401 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:11.425642 17358 external_mini_cluster.cc:1428] Setting key 6418d6ceab6355457869156f977a92f5
I20260618 22:31:11.430895 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a: Bootstrap starting.
I20260618 22:31:11.433528 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:11.434281 5879 log.cc:824] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:31:11.436509 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a: No bootstrap required, opened a new log
I20260618 22:31:11.440680 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:11.427036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:48721 (local address 127.0.0.1:46331)
0618 22:31:11.427493 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:11.427513 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:11.427560 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:31:11.428353 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:11.428376 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:11.428837 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:11.429227 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:11.429889 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:11.431332 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:11.432212 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:11.432902 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:11.433068 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:11.435576 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:11.435583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:11.435600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:11.435631 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:11.437445 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:11.437980 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:11.437986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:11.437993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:11.438135 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:11.438405 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:11.438408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:11.438410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:11.438992 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:11.439199 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:11.439433 (+ 234us) server_negotiation.cc:299] Negotiation successful
0618 22:31:11.439676 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260618 22:31:11.441045 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } }
I20260618 22:31:11.441270 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:11.441314 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807580e729b84f4c8ec9c540a1d5e12a, State: Initialized, Role: FOLLOWER
I20260618 22:31:11.441740 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } }
I20260618 22:31:11.441867 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:11.441931 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:11.442039 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:11.443159 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } }
I20260618 22:31:11.443491 5879 leader_election.cc:302] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807580e729b84f4c8ec9c540a1d5e12a; no voters:
I20260618 22:31:11.443728 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:11.443816 5879 leader_election.cc:288] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:11.443902 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:11.444077 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [term 1 LEADER]: Becoming Leader. State: Replica: 807580e729b84f4c8ec9c540a1d5e12a, State: Running, Role: LEADER
I20260618 22:31:11.444386 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } }
I20260618 22:31:11.445182 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:11.445839 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807580e729b84f4c8ec9c540a1d5e12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } } }
I20260618 22:31:11.445957 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:11.446314 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807580e729b84f4c8ec9c540a1d5e12a. Latest consensus state: current_term: 1 leader_uuid: "807580e729b84f4c8ec9c540a1d5e12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807580e729b84f4c8ec9c540a1d5e12a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46331 } } }
I20260618 22:31:11.446396 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:11.446738 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:11.449945 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:11.456044 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:11.446081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:48739 (local address 127.0.0.1:46331)
0618 22:31:11.446195 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:11.446201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:11.447340 (+ 1139us) server_negotiation.cc:407] Connection header received
0618 22:31:11.447536 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:11.447540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:11.447599 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:11.447682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:11.448471 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:11.449050 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:11.450204 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:11.450532 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:11.450643 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:11.452448 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:11.452452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:11.452455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:11.452484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:11.454178 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:11.454776 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:11.454779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:11.454781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:11.454870 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:11.455178 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:11.455182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:11.455184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:11.455412 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:11.455555 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:11.455777 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 22:31:11.455881 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:31:11.461824 5896 catalog_manager.cc:1347] Generated new cluster ID: 53a2461e90b240b0bf454ef799812288
I20260618 22:31:11.461920 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:11.473699 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:11.475014 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:11.481676 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807580e729b84f4c8ec9c540a1d5e12a: Generated new TSK 0
I20260618 22:31:11.482369 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:11.486940 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5811
W20260618 22:31:11.491274 5796 connection.cc:535] client connection to 127.0.0.1:46331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46331): Connection reset by peer (error 104)
2026-06-18T22:31:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5910](info): set up 2 sockets
Jun 18 22:31:11 dist-test-slave-jm5g krb5kdc[5910](info): commencing operation
krb5kdc: starting...
W20260618 22:31:12.438413 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:12 dist-test-slave-jm5g krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:12.549327 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:12.659225 5923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:12.659525 5923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:12.659569 5923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:12.663708 5923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:12.663784 5923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:12.663805 5923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:12.663820 5923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:12.663834 5923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:12.669250 5923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:12.670317 5923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:12.671392 5923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:12.679911 5931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:12.680032 5923 server_base.cc:1034] running on GCE node
W20260618 22:31:12.679878 5928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:12.680223 5929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:12.680780 5923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:12.681964 5923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:12.683158 5923 hybrid_clock.cc:648] HybridClock initialized: now 1781821872683135 us; error 39 us; skew 500 ppm
Jun 18 22:31:12 dist-test-slave-jm5g krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:12.686033 5923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:12.687018 5923 webserver.cc:460] Webserver started at http://127.0.0.1:44801/ using document root <none> and password file <none>
I20260618 22:31:12.687574 5923 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:12.687646 5923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:12.687847 5923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:12.689918 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "65b2d816f01c452ebfa168ad6525f63c"
format_stamp: "Formatted at 2026-06-18 22:31:12 on dist-test-slave-jm5g"
server_key: "df3818daf5103fcff8d353f96358d702"
server_key_iv: "b1928adfb75fc80f45e0b293e45ce0b3"
server_key_version: "encryptionkey@0"
I20260618 22:31:12.690469 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "65b2d816f01c452ebfa168ad6525f63c"
format_stamp: "Formatted at 2026-06-18 22:31:12 on dist-test-slave-jm5g"
server_key: "df3818daf5103fcff8d353f96358d702"
server_key_iv: "b1928adfb75fc80f45e0b293e45ce0b3"
server_key_version: "encryptionkey@0"
I20260618 22:31:12.694183 5923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:12.696750 5938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:12.697997 5923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:12.698140 5923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "65b2d816f01c452ebfa168ad6525f63c"
format_stamp: "Formatted at 2026-06-18 22:31:12 on dist-test-slave-jm5g"
server_key: "df3818daf5103fcff8d353f96358d702"
server_key_iv: "b1928adfb75fc80f45e0b293e45ce0b3"
server_key_version: "encryptionkey@0"
I20260618 22:31:12.698246 5923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:12.744274 5923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:12.747833 5923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:12.748050 5923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:12.754920 5923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44147
I20260618 22:31:12.754954 5990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44147 every 8 connection(s)
I20260618 22:31:12.755956 5923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:12.758821 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:12.764484 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: Bootstrap starting.
I20260618 22:31:12.765826 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 5923
I20260618 22:31:12.765954 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:12.766188 17358 external_mini_cluster.cc:1428] Setting key f51232f0df3a15e5d2f979d34972fd28
I20260618 22:31:12.767207 5991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:12.768047 5991 log.cc:824] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:12.770047 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: No bootstrap required, opened a new log
I20260618 22:31:12.773793 5991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260618 22:31:12.774039 5991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:12.774116 5991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b2d816f01c452ebfa168ad6525f63c, State: Initialized, Role: FOLLOWER
I20260618 22:31:12.774636 5991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260618 22:31:12.774782 5991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:12.774847 5991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:12.774938 5991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:12.775169 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:12.767349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:53689 (local address 127.0.0.1:44147)
0618 22:31:12.767807 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:12.767823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:12.767859 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:12.768653 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:12.768679 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:12.769071 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:12.769392 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:12.769809 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:12.770842 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:12.771684 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:12.772193 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:12.772342 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:12.773312 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:12.773318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:12.773333 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:12.773755 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:12.773810 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:12.773832 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:12.773949 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:12.774128 (+ 179us) server_negotiation.cc:299] Negotiation successful
0618 22:31:12.774263 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260618 22:31:12.775910 5991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260618 22:31:12.776270 5991 leader_election.cc:302] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b2d816f01c452ebfa168ad6525f63c; no voters:
I20260618 22:31:12.776623 5991 leader_election.cc:288] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:12.776731 5996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:12.776957 5996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [term 1 LEADER]: Becoming Leader. State: Replica: 65b2d816f01c452ebfa168ad6525f63c, State: Running, Role: LEADER
I20260618 22:31:12.777271 5996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } }
I20260618 22:31:12.777897 5991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:12.778258 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:12.778909 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b2d816f01c452ebfa168ad6525f63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } } }
I20260618 22:31:12.779026 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:12.779316 5998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b2d816f01c452ebfa168ad6525f63c. Latest consensus state: current_term: 1 leader_uuid: "65b2d816f01c452ebfa168ad6525f63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b2d816f01c452ebfa168ad6525f63c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44147 } } }
I20260618 22:31:12.779392 5998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:12.783392 6015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:12.783531 6015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:12.783838 6005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:12.784786 6005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:12.785185 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:12.780362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45775 (local address 127.0.0.1:44147)
0618 22:31:12.780530 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:12.780533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:12.780685 (+ 152us) server_negotiation.cc:407] Connection header received
0618 22:31:12.780869 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:12.780872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:12.780920 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:12.781001 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:12.781356 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:12.782091 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:12.783329 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:12.783646 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:12.783738 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:12.784668 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:12.784673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:12.784676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:12.784692 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:12.784722 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:12.784736 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:12.784823 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:12.785012 (+ 189us) server_negotiation.cc:299] Negotiation successful
0618 22:31:12.785051 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260618 22:31:12.792097 6005 catalog_manager.cc:1347] Generated new cluster ID: 4d196e3053c64ddcadb6f61f5807cd50
I20260618 22:31:12.792196 6005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:12.810084 6005 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:12.811201 6005 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:12.822010 6005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b2d816f01c452ebfa168ad6525f63c: Generated new TSK 0
I20260618 22:31:12.822964 6005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:12.832867 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 5923
W20260618 22:31:12.837364 5908 connection.cc:535] client connection to 127.0.0.1:44147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44147): Connection reset by peer (error 104)
2026-06-18T22:31:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:12 dist-test-slave-jm5g krb5kdc[6022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:12 dist-test-slave-jm5g krb5kdc[6022](info): set up 2 sockets
Jun 18 22:31:12 dist-test-slave-jm5g krb5kdc[6022](info): commencing operation
krb5kdc: starting...
W20260618 22:31:13.782423 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:13 dist-test-slave-jm5g krb5kdc[6022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:13.894773 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:14.002457 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:14.002737 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:14.002795 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:14.006734 6035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:14.006799 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:14.006825 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:14.006847 6035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:14.006875 6035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:14.012177 6035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:14.013252 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:14.014273 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:14.022511 6035 server_base.cc:1034] running on GCE node
W20260618 22:31:14.022413 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:14.022442 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:14.022424 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:14.023239 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:14.024299 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:14.025449 6035 hybrid_clock.cc:648] HybridClock initialized: now 1781821874025434 us; error 28 us; skew 500 ppm
Jun 18 22:31:14 dist-test-slave-jm5g krb5kdc[6022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:14.028087 6035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:14.028963 6035 webserver.cc:460] Webserver started at http://127.0.0.1:41645/ using document root <none> and password file <none>
I20260618 22:31:14.029492 6035 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:14.029556 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:14.029742 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:14.031806 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "c7f58b3c380e459d9120e66245299ef0"
format_stamp: "Formatted at 2026-06-18 22:31:14 on dist-test-slave-jm5g"
server_key: "3beed897bdae2603f1a46da1ec5aa0cb"
server_key_iv: "9ae04b7e9b03a39497dbab05b36073d5"
server_key_version: "encryptionkey@0"
I20260618 22:31:14.032346 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "c7f58b3c380e459d9120e66245299ef0"
format_stamp: "Formatted at 2026-06-18 22:31:14 on dist-test-slave-jm5g"
server_key: "3beed897bdae2603f1a46da1ec5aa0cb"
server_key_iv: "9ae04b7e9b03a39497dbab05b36073d5"
server_key_version: "encryptionkey@0"
I20260618 22:31:14.035946 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:31:14.038311 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:14.039660 6035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:14.039777 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "c7f58b3c380e459d9120e66245299ef0"
format_stamp: "Formatted at 2026-06-18 22:31:14 on dist-test-slave-jm5g"
server_key: "3beed897bdae2603f1a46da1ec5aa0cb"
server_key_iv: "9ae04b7e9b03a39497dbab05b36073d5"
server_key_version: "encryptionkey@0"
I20260618 22:31:14.039875 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:14.044436 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:14.047240 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:14.047405 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:14.054075 6035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37917
I20260618 22:31:14.054086 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37917 every 8 connection(s)
I20260618 22:31:14.055053 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:14.057804 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:14.060488 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6035
I20260618 22:31:14.060613 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:14.060851 17358 external_mini_cluster.cc:1428] Setting key 11c4f2bd97840c29db8e478bc6708ae1
I20260618 22:31:14.063601 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0: Bootstrap starting.
I20260618 22:31:14.065738 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:14.062101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:51375 (local address 127.0.0.1:37917)
0618 22:31:14.062493 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:14.062507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:14.062542 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:14.063207 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:14.063226 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:14.063525 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:14.063838 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:14.064595 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:14.064600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:14.064611 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:14.064621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:14.064674 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:14.064693 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:14.064810 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:14.064903 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 22:31:14.065071 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260618 22:31:14.065970 6103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:14.066751 6103 log.cc:824] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:14.068447 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:14.068821 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0: No bootstrap required, opened a new log
I20260618 22:31:14.071692 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:14.070279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:33625 (local address 127.0.0.1:37917)
0618 22:31:14.070371 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:14.070374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:14.070512 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:31:14.070650 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:14.070653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:14.070681 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:14.070741 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:14.071369 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:14.071371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:14.071373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:14.071382 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:14.071404 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:14.071413 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:14.071481 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:14.071564 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 22:31:14.071593 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 22:31:14.072889 6103 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } }
I20260618 22:31:14.073108 6103 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:14.073161 6103 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f58b3c380e459d9120e66245299ef0, State: Initialized, Role: FOLLOWER
I20260618 22:31:14.073554 6103 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } }
I20260618 22:31:14.073678 6103 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:14.073740 6103 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:14.073829 6103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:14.074765 6103 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } }
I20260618 22:31:14.075078 6103 leader_election.cc:302] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f58b3c380e459d9120e66245299ef0; no voters:
I20260618 22:31:14.075354 6103 leader_election.cc:288] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:14.075480 6113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:14.075670 6113 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [term 1 LEADER]: Becoming Leader. State: Replica: c7f58b3c380e459d9120e66245299ef0, State: Running, Role: LEADER
I20260618 22:31:14.076004 6113 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } }
I20260618 22:31:14.076488 6103 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:14.077584 6115 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f58b3c380e459d9120e66245299ef0. Latest consensus state: current_term: 1 leader_uuid: "c7f58b3c380e459d9120e66245299ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } } }
I20260618 22:31:14.077687 6115 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:14.077943 6114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f58b3c380e459d9120e66245299ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f58b3c380e459d9120e66245299ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37917 } } }
I20260618 22:31:14.078018 6114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:14.078878 6122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:14.081277 6122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:14.086369 6122 catalog_manager.cc:1347] Generated new cluster ID: bd697612b6ed479089cf7016de6c2143
I20260618 22:31:14.086424 6122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:14.112449 6122 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:14.112767 6122 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:14.124106 6122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f58b3c380e459d9120e66245299ef0: Generated new TSK 0
I20260618 22:31:14.124801 6122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:14.128702 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6035
2026-06-18T22:31:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:14 dist-test-slave-jm5g krb5kdc[6134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:14 dist-test-slave-jm5g krb5kdc[6134](info): set up 2 sockets
Jun 18 22:31:14 dist-test-slave-jm5g krb5kdc[6134](info): commencing operation
krb5kdc: starting...
W20260618 22:31:15.074525 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:15.186923 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:15.294724 6147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:15.295014 6147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:15.295074 6147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:15.299093 6147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:15.299163 6147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:15.299188 6147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:15.299211 6147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:15.299242 6147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:15.304580 6147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:15.305651 6147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:15.306705 6147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:15.314869 6147 server_base.cc:1034] running on GCE node
W20260618 22:31:15.314821 6153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:15.314836 6155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:15.314826 6152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:15.315615 6147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:15.316707 6147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:15.317860 6147 hybrid_clock.cc:648] HybridClock initialized: now 1781821875317836 us; error 40 us; skew 500 ppm
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:15.324596 6147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:15.325578 6147 webserver.cc:460] Webserver started at http://127.0.0.1:37303/ using document root <none> and password file <none>
I20260618 22:31:15.326133 6147 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:15.326198 6147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:15.326400 6147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:15.328531 6147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "8ac899d86f4641d6adac14ae44613c12"
format_stamp: "Formatted at 2026-06-18 22:31:15 on dist-test-slave-jm5g"
server_key: "0d0a05a9ee03df6e6ddf510ebc9fdb12"
server_key_iv: "25df25fde6eecd9905784be35498877f"
server_key_version: "encryptionkey@0"
I20260618 22:31:15.329110 6147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "8ac899d86f4641d6adac14ae44613c12"
format_stamp: "Formatted at 2026-06-18 22:31:15 on dist-test-slave-jm5g"
server_key: "0d0a05a9ee03df6e6ddf510ebc9fdb12"
server_key_iv: "25df25fde6eecd9905784be35498877f"
server_key_version: "encryptionkey@0"
I20260618 22:31:15.332845 6147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260618 22:31:15.335218 6162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:15.336397 6147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:15.336512 6147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "8ac899d86f4641d6adac14ae44613c12"
format_stamp: "Formatted at 2026-06-18 22:31:15 on dist-test-slave-jm5g"
server_key: "0d0a05a9ee03df6e6ddf510ebc9fdb12"
server_key_iv: "25df25fde6eecd9905784be35498877f"
server_key_version: "encryptionkey@0"
I20260618 22:31:15.336611 6147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:15.347990 6147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:15.350737 6147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:15.350911 6147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:15.357560 6147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42555
I20260618 22:31:15.357577 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42555 every 8 connection(s)
I20260618 22:31:15.358490 6147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:15.361289 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:15.362468 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6147
I20260618 22:31:15.362624 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:15.362916 17358 external_mini_cluster.cc:1428] Setting key 27202f83c429f54447f57b2496b5f138
I20260618 22:31:15.367318 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12: Bootstrap starting.
I20260618 22:31:15.369493 6215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:15.370296 6215 log.cc:824] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:31:15.372309 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12: No bootstrap required, opened a new log
I20260618 22:31:15.375876 6215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260618 22:31:15.376097 6215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:15.376134 6215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac899d86f4641d6adac14ae44613c12, State: Initialized, Role: FOLLOWER
I20260618 22:31:15.376448 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:15.364351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:37679 (local address 127.0.0.1:42555)
0618 22:31:15.364778 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:15.364792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:15.364828 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:15.365446 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:15.365464 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:15.365823 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:15.366137 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:15.366933 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:15.367928 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:15.368786 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:15.369243 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:15.369381 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:15.372055 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:15.372059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:15.372071 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:15.372097 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:15.373612 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:15.374087 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:15.374092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:15.374096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:15.374204 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:15.374512 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:15.374515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:15.374516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:15.374926 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:15.375130 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:15.375454 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 22:31:15.375669 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
I20260618 22:31:15.376541 6215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260618 22:31:15.376745 6215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:15.376811 6215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:15.376909 6215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:15.378036 6215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260618 22:31:15.378403 6215 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac899d86f4641d6adac14ae44613c12; no voters:
I20260618 22:31:15.378767 6215 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:15.378880 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:15.379087 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac899d86f4641d6adac14ae44613c12, State: Running, Role: LEADER
I20260618 22:31:15.379451 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260618 22:31:15.379639 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:15.380060 6215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:15.382607 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac899d86f4641d6adac14ae44613c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260618 22:31:15.382747 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:15.383113 6240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:15.383016 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac899d86f4641d6adac14ae44613c12. Latest consensus state: current_term: 1 leader_uuid: "8ac899d86f4641d6adac14ae44613c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac899d86f4641d6adac14ae44613c12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260618 22:31:15.383493 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:15.386353 6240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:15.390946 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:15.382631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:34659 (local address 127.0.0.1:42555)
0618 22:31:15.382723 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:15.382727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:15.382741 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:31:15.382780 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:15.382783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:15.382829 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:15.382900 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:15.383825 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:15.384583 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:15.385653 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:15.385974 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:15.386077 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:15.387740 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:15.387744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:15.387747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:15.387772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:15.389203 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:15.389653 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:15.389657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:15.389659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:15.389732 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:15.390021 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:15.390024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:15.390026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:15.390199 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:15.390324 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:15.390693 (+ 369us) server_negotiation.cc:299] Negotiation successful
0618 22:31:15.390796 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 22:31:15.392989 6240 catalog_manager.cc:1347] Generated new cluster ID: 0a5ac7fa9d9f44a482c0bdc89774a6d6
I20260618 22:31:15.393054 6240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:15.408425 6240 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:15.409727 6240 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:15.414968 6240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac899d86f4641d6adac14ae44613c12: Generated new TSK 0
I20260618 22:31:15.415786 6240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:15.419066 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6147
W20260618 22:31:15.423354 6132 connection.cc:535] client connection to 127.0.0.1:42555 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42555): Connection reset by peer (error 104)
2026-06-18T22:31:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6246](info): set up 2 sockets
Jun 18 22:31:15 dist-test-slave-jm5g krb5kdc[6246](info): commencing operation
krb5kdc: starting...
W20260618 22:31:16.370572 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:16 dist-test-slave-jm5g krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:16.482300 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:16.594903 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:16.595193 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:16.595237 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:16.599305 6259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:16.599377 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:16.599395 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:16.599412 6259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:16.599426 6259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:16.604872 6259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:16.605964 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:16.607051 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:16.615263 6264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:16.615285 6265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:16.615268 6267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:16.615450 6259 server_base.cc:1034] running on GCE node
I20260618 22:31:16.616143 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:16.617272 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:16.618429 6259 hybrid_clock.cc:648] HybridClock initialized: now 1781821876618415 us; error 34 us; skew 500 ppm
Jun 18 22:31:16 dist-test-slave-jm5g krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:16.621160 6259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:16.622082 6259 webserver.cc:460] Webserver started at http://127.0.0.1:46443/ using document root <none> and password file <none>
I20260618 22:31:16.622653 6259 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:16.622718 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:16.622915 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:16.624987 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "c2002ca5385b408785ec8294efe716b2"
format_stamp: "Formatted at 2026-06-18 22:31:16 on dist-test-slave-jm5g"
server_key: "ceee98e38d593550351d8f7c0804c22a"
server_key_iv: "ba749c8d46890676f4448e5ca5e228b4"
server_key_version: "encryptionkey@0"
I20260618 22:31:16.625516 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "c2002ca5385b408785ec8294efe716b2"
format_stamp: "Formatted at 2026-06-18 22:31:16 on dist-test-slave-jm5g"
server_key: "ceee98e38d593550351d8f7c0804c22a"
server_key_iv: "ba749c8d46890676f4448e5ca5e228b4"
server_key_version: "encryptionkey@0"
I20260618 22:31:16.629302 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260618 22:31:16.631711 6274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:16.632933 6259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:31:16.633057 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "c2002ca5385b408785ec8294efe716b2"
format_stamp: "Formatted at 2026-06-18 22:31:16 on dist-test-slave-jm5g"
server_key: "ceee98e38d593550351d8f7c0804c22a"
server_key_iv: "ba749c8d46890676f4448e5ca5e228b4"
server_key_version: "encryptionkey@0"
I20260618 22:31:16.633158 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:16.644757 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:16.647917 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:16.648104 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:16.655241 6259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34017
I20260618 22:31:16.655254 6326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34017 every 8 connection(s)
I20260618 22:31:16.656226 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:16.658169 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6259
I20260618 22:31:16.658303 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:16.658615 17358 external_mini_cluster.cc:1428] Setting key e4c4b2c9a7731f7a1f37a556222ee800
I20260618 22:31:16.659335 6327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:16.665666 6327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2: Bootstrap starting.
I20260618 22:31:16.667524 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:16.659918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:34321 (local address 127.0.0.1:34017)
0618 22:31:16.660334 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:16.660349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:16.660384 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:16.661040 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:16.661058 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:16.661403 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:16.661692 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:16.662005 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:16.663043 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:16.664191 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:16.664666 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:16.664775 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:16.665711 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:16.665716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:16.665730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:16.666157 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:16.666212 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:16.666235 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:16.666341 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:16.666627 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:31:16.666780 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260618 22:31:16.667989 6327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:16.668766 6327 log.cc:824] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:16.670435 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:16.670964 6327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2: No bootstrap required, opened a new log
I20260618 22:31:16.675510 6327 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260618 22:31:16.675800 6327 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:16.675864 6327 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2002ca5385b408785ec8294efe716b2, State: Initialized, Role: FOLLOWER
I20260618 22:31:16.676370 6327 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260618 22:31:16.676534 6327 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:16.676550 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:16.672606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:37315 (local address 127.0.0.1:34017)
0618 22:31:16.672727 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:16.672731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:16.672890 (+ 159us) server_negotiation.cc:407] Connection header received
0618 22:31:16.673014 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:16.673016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:16.673055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:16.673116 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:16.673401 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:16.673903 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:16.674909 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:16.675214 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:16.675313 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:16.676131 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:16.676135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:16.676138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:16.676151 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:16.676177 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:16.676190 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:16.676268 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:16.676360 (+ 92us) server_negotiation.cc:299] Negotiation successful
0618 22:31:16.676397 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:31:16.676612 6327 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:16.676707 6327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:16.677623 6327 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260618 22:31:16.677945 6327 leader_election.cc:302] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2002ca5385b408785ec8294efe716b2; no voters:
I20260618 22:31:16.678241 6327 leader_election.cc:288] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:16.678326 6337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:16.678578 6337 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [term 1 LEADER]: Becoming Leader. State: Replica: c2002ca5385b408785ec8294efe716b2, State: Running, Role: LEADER
I20260618 22:31:16.678920 6337 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } }
I20260618 22:31:16.679415 6327 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:16.680423 6338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2002ca5385b408785ec8294efe716b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } } }
I20260618 22:31:16.680560 6338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:16.680899 6346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:16.680879 6339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2002ca5385b408785ec8294efe716b2. Latest consensus state: current_term: 1 leader_uuid: "c2002ca5385b408785ec8294efe716b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2002ca5385b408785ec8294efe716b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34017 } } }
I20260618 22:31:16.681010 6339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:16.683745 6346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:16.689003 6346 catalog_manager.cc:1347] Generated new cluster ID: db139d03804f49dfa66b97121645ac29
I20260618 22:31:16.689069 6346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:16.742861 6346 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:16.744024 6346 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:16.753997 6346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2002ca5385b408785ec8294efe716b2: Generated new TSK 0
I20260618 22:31:16.754719 6346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:16.761059 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6259
W20260618 22:31:16.765457 6244 connection.cc:535] client connection to 127.0.0.1:34017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34017): Connection reset by peer (error 104)
2026-06-18T22:31:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:16 dist-test-slave-jm5g krb5kdc[6358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:16 dist-test-slave-jm5g krb5kdc[6358](info): set up 2 sockets
Jun 18 22:31:16 dist-test-slave-jm5g krb5kdc[6358](info): commencing operation
krb5kdc: starting...
W20260618 22:31:17.718478 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:17 dist-test-slave-jm5g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:31:17.827277 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:17.933836 6371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:17.934120 6371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:17.934180 6371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:17.938239 6371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:17.938310 6371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:17.938337 6371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:17.938360 6371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:17.938390 6371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:17.943732 6371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:17.944796 6371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:17.945803 6371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:17.953939 6371 server_base.cc:1034] running on GCE node
W20260618 22:31:17.953948 6376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:17.953908 6379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:17.953918 6377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:17.954777 6371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:17.955885 6371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:17.957052 6371 hybrid_clock.cc:648] HybridClock initialized: now 1781821877957031 us; error 36 us; skew 500 ppm
Jun 18 22:31:17 dist-test-slave-jm5g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:17.959789 6371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:31:17.960881 6371 webserver.cc:460] Webserver started at http://127.0.0.1:45661/ using document root <none> and password file <none>
I20260618 22:31:17.961587 6371 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:17.961670 6371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:17.961897 6371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:17.964108 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "5a55cc5645674814971636bf841c76ec"
format_stamp: "Formatted at 2026-06-18 22:31:17 on dist-test-slave-jm5g"
server_key: "38251efe40ab81d72b6db1ca2a216a72"
server_key_iv: "e233b77a2a1e9dbac20ee597f5c355a4"
server_key_version: "encryptionkey@0"
I20260618 22:31:17.964634 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "5a55cc5645674814971636bf841c76ec"
format_stamp: "Formatted at 2026-06-18 22:31:17 on dist-test-slave-jm5g"
server_key: "38251efe40ab81d72b6db1ca2a216a72"
server_key_iv: "e233b77a2a1e9dbac20ee597f5c355a4"
server_key_version: "encryptionkey@0"
I20260618 22:31:17.968243 6371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260618 22:31:17.970531 6386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:17.971733 6371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:31:17.971843 6371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "5a55cc5645674814971636bf841c76ec"
format_stamp: "Formatted at 2026-06-18 22:31:17 on dist-test-slave-jm5g"
server_key: "38251efe40ab81d72b6db1ca2a216a72"
server_key_iv: "e233b77a2a1e9dbac20ee597f5c355a4"
server_key_version: "encryptionkey@0"
I20260618 22:31:17.971935 6371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:17.976332 6371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:17.978996 6371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:17.979151 6371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:17.985564 6371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33133
I20260618 22:31:17.985592 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33133 every 8 connection(s)
I20260618 22:31:17.986449 6371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:17.989091 6439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:17.992992 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6371
I20260618 22:31:17.993108 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:17.993353 17358 external_mini_cluster.cc:1428] Setting key 120f34d46a81abfd01479be0000b4058
I20260618 22:31:17.994736 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: Bootstrap starting.
I20260618 22:31:17.997432 6439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:17.998370 6439 log.cc:824] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:17.998795 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:17.994744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:38403 (local address 127.0.0.1:33133)
0618 22:31:17.995173 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:17.995190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:17.995230 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:31:17.996045 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:17.996070 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:17.996448 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:17.996795 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:17.997428 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:17.997433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:17.997448 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:17.997459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:17.997515 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:17.997541 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:17.997647 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:17.997720 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 22:31:17.997873 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260618 22:31:18.000353 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: No bootstrap required, opened a new log
I20260618 22:31:18.001489 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:18.004207 6439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260618 22:31:18.004431 6439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:18.004487 6439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55cc5645674814971636bf841c76ec, State: Initialized, Role: FOLLOWER
I20260618 22:31:18.004829 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:18.003388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:36895 (local address 127.0.0.1:33133)
0618 22:31:18.003508 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:18.003512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:18.003649 (+ 137us) server_negotiation.cc:407] Connection header received
0618 22:31:18.003752 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:18.003755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:18.003784 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:18.003843 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:18.004490 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:18.004493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:18.004496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:18.004508 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:18.004536 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:18.004549 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:18.004619 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:18.004688 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 22:31:18.004724 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 22:31:18.004943 6439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260618 22:31:18.005102 6439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:18.005172 6439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:18.005266 6439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:18.006172 6439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260618 22:31:18.006479 6439 leader_election.cc:302] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a55cc5645674814971636bf841c76ec; no voters:
I20260618 22:31:18.006788 6439 leader_election.cc:288] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:18.006863 6449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:18.007066 6449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [term 1 LEADER]: Becoming Leader. State: Replica: 5a55cc5645674814971636bf841c76ec, State: Running, Role: LEADER
I20260618 22:31:18.007372 6449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } }
I20260618 22:31:18.007874 6439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:18.008849 6451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a55cc5645674814971636bf841c76ec. Latest consensus state: current_term: 1 leader_uuid: "5a55cc5645674814971636bf841c76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } } }
I20260618 22:31:18.008966 6451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:18.009232 6450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a55cc5645674814971636bf841c76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a55cc5645674814971636bf841c76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33133 } } }
I20260618 22:31:18.009312 6450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:18.012171 6464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:18.012234 6464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:18.012307 6458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:18.013087 6458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:18.018151 6458 catalog_manager.cc:1347] Generated new cluster ID: ae6b7ccbea7549bcbd85a02f1113a2c1
I20260618 22:31:18.018216 6458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:18.024511 6458 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:18.024842 6458 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:18.029345 6458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a55cc5645674814971636bf841c76ec: Generated new TSK 0
I20260618 22:31:18.030130 6458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:18.037395 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6371
2026-06-18T22:31:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:18 dist-test-slave-jm5g krb5kdc[6470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:18 dist-test-slave-jm5g krb5kdc[6470](info): set up 2 sockets
Jun 18 22:31:18 dist-test-slave-jm5g krb5kdc[6470](info): commencing operation
krb5kdc: starting...
W20260618 22:31:18.998538 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:19.106909 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34087
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:34087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:19.215025 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:19.215293 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:19.215333 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:19.219326 6483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:19.219398 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:19.219419 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:19.219434 6483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:19.219449 6483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:19.224728 6483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:34087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:34087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:19.225772 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:19.226807 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:19.234860 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:19.234865 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:19.234941 6483 server_base.cc:1034] running on GCE node
W20260618 22:31:19.235098 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:19.235594 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:19.236675 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:19.237841 6483 hybrid_clock.cc:648] HybridClock initialized: now 1781821879237814 us; error 38 us; skew 500 ppm
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821879, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:19.240412 6483 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:19.241261 6483 webserver.cc:460] Webserver started at http://127.16.243.190:35493/ using document root <none> and password file <none>
I20260618 22:31:19.241799 6483 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:19.241863 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:19.242071 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:19.244143 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "f059384dd08a4825aa51bd7ef7fdd367"
format_stamp: "Formatted at 2026-06-18 22:31:19 on dist-test-slave-jm5g"
server_key: "455a0409093e0fed2fbe7773ca988d33"
server_key_iv: "20a4a1bb84f28e2f73292a3aa9edbacb"
server_key_version: "encryptionkey@0"
I20260618 22:31:19.244656 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "f059384dd08a4825aa51bd7ef7fdd367"
format_stamp: "Formatted at 2026-06-18 22:31:19 on dist-test-slave-jm5g"
server_key: "455a0409093e0fed2fbe7773ca988d33"
server_key_iv: "20a4a1bb84f28e2f73292a3aa9edbacb"
server_key_version: "encryptionkey@0"
I20260618 22:31:19.248306 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260618 22:31:19.250654 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:19.251848 6483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:19.251971 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "f059384dd08a4825aa51bd7ef7fdd367"
format_stamp: "Formatted at 2026-06-18 22:31:19 on dist-test-slave-jm5g"
server_key: "455a0409093e0fed2fbe7773ca988d33"
server_key_iv: "20a4a1bb84f28e2f73292a3aa9edbacb"
server_key_version: "encryptionkey@0"
I20260618 22:31:19.252067 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:19.269924 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:19.273041 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:19.273236 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:19.280184 6483 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:34087
I20260618 22:31:19.280212 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:34087 every 8 connection(s)
I20260618 22:31:19.281164 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:19.282387 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6483
I20260618 22:31:19.282519 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:19.282790 17358 external_mini_cluster.cc:1428] Setting key 6f702e23231425c705945d59e0b2a719
I20260618 22:31:19.284127 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:19.290510 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: Bootstrap starting.
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:19.292953 6551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:19.293635 6551 log.cc:824] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:19.295652 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: No bootstrap required, opened a new log
I20260618 22:31:19.296933 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:19.284124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38249 (local address 127.16.243.190:34087)
0618 22:31:19.284519 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:19.284535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:19.284570 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:19.285270 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:19.285289 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:19.285631 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:19.285907 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:19.286793 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:19.287785 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:19.288951 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:19.289427 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:19.289535 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:19.292260 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:19.292265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:19.292278 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:19.292305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:19.293904 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:19.294394 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:19.294399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:19.294403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:19.294512 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:19.294855 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:19.294859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:19.294860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:19.295337 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:19.295550 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:19.295799 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:31:19.296025 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":122,"threads_started":1}
I20260618 22:31:19.299401 6551 raft_consensus.cc:348] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } }
I20260618 22:31:19.299638 6551 raft_consensus.cc:374] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:19.299695 6551 raft_consensus.cc:729] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f059384dd08a4825aa51bd7ef7fdd367, State: Initialized, Role: FOLLOWER
I20260618 22:31:19.299787 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:19.300196 6551 consensus_queue.cc:260] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } }
I20260618 22:31:19.300366 6551 raft_consensus.cc:388] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:19.300428 6551 raft_consensus.cc:482] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:19.300521 6551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:19.301643 6551 raft_consensus.cc:504] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } }
I20260618 22:31:19.302021 6551 leader_election.cc:302] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f059384dd08a4825aa51bd7ef7fdd367; no voters:
I20260618 22:31:19.302346 6551 leader_election.cc:288] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:19.302438 6561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:19.302647 6561 raft_consensus.cc:686] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [term 1 LEADER]: Becoming Leader. State: Replica: f059384dd08a4825aa51bd7ef7fdd367, State: Running, Role: LEADER
I20260618 22:31:19.302986 6561 consensus_queue.cc:237] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } }
I20260618 22:31:19.303678 6551 sys_catalog.cc:564] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:19.304212 6561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f059384dd08a4825aa51bd7ef7fdd367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } } }
I20260618 22:31:19.304908 6561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:19.304797 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f059384dd08a4825aa51bd7ef7fdd367. Latest consensus state: current_term: 1 leader_uuid: "f059384dd08a4825aa51bd7ef7fdd367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059384dd08a4825aa51bd7ef7fdd367" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 34087 } } }
I20260618 22:31:19.305189 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367 [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:19.308382 6576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:19.308460 6576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:19.308547 6570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:19.309314 6570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:19.310051 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:19.301868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54959 (local address 127.16.243.190:34087)
0618 22:31:19.301989 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:19.301993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:19.302161 (+ 168us) server_negotiation.cc:407] Connection header received
0618 22:31:19.302257 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:19.302259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:19.302297 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:19.302365 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:19.303128 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:19.303684 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:19.304610 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:19.304917 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:19.304993 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:19.306620 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:19.306623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:19.306626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:19.306650 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:19.308210 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:19.308739 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:19.308743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:19.308745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:19.308828 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:19.309182 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:19.309185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:19.309187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:19.309391 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:19.309534 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:19.309815 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:31:19.309911 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:31:19.314347 6570 catalog_manager.cc:1347] Generated new cluster ID: 6fff6e2c893c47b89944d13c2ccb7e81
I20260618 22:31:19.314412 6570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:19.332604 6570 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:19.333514 6570 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:19.342773 6570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f059384dd08a4825aa51bd7ef7fdd367: Generated new TSK 0
I20260618 22:31:19.343459 6570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:19.354398 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6483
W20260618 22:31:19.358979 6468 connection.cc:535] client connection to 127.16.243.190:34087 recv error: Network error: failed to read from TLS socket (remote: 127.16.243.190:34087): Connection reset by peer (error 104)
2026-06-18T22:31:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6582](info): set up 2 sockets
Jun 18 22:31:19 dist-test-slave-jm5g krb5kdc[6582](info): commencing operation
krb5kdc: starting...
W20260618 22:31:20.314435 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:20 dist-test-slave-jm5g krb5kdc[6582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:20.425527 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33211
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:20.535266 6595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:20.535553 6595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:20.535617 6595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:20.539754 6595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:20.539824 6595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:20.539849 6595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:20.539872 6595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:20.539901 6595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:20.545359 6595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:33211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:20.546456 6595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:20.547492 6595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:20.555805 6600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:20.555840 6603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:20.555848 6601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:20.555979 6595 server_base.cc:1034] running on GCE node
I20260618 22:31:20.556704 6595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:20.557793 6595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:20.558934 6595 hybrid_clock.cc:648] HybridClock initialized: now 1781821880558911 us; error 46 us; skew 500 ppm
Jun 18 22:31:20 dist-test-slave-jm5g krb5kdc[6582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:20.568863 6595 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:20.569821 6595 webserver.cc:460] Webserver started at http://127.16.243.190:41993/ using document root <none> and password file <none>
I20260618 22:31:20.570372 6595 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:20.570432 6595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:20.570654 6595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:20.572736 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "6b11894310694bad95c9a0af4c8954bc"
format_stamp: "Formatted at 2026-06-18 22:31:20 on dist-test-slave-jm5g"
server_key: "d0fb281a5da307a688c8a7d7a6c6346a"
server_key_iv: "f693f066548a6398b2052503cd63d044"
server_key_version: "encryptionkey@0"
I20260618 22:31:20.573251 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "6b11894310694bad95c9a0af4c8954bc"
format_stamp: "Formatted at 2026-06-18 22:31:20 on dist-test-slave-jm5g"
server_key: "d0fb281a5da307a688c8a7d7a6c6346a"
server_key_iv: "f693f066548a6398b2052503cd63d044"
server_key_version: "encryptionkey@0"
I20260618 22:31:20.576967 6595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:31:20.579411 6610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:20.580673 6595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:20.580780 6595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "6b11894310694bad95c9a0af4c8954bc"
format_stamp: "Formatted at 2026-06-18 22:31:20 on dist-test-slave-jm5g"
server_key: "d0fb281a5da307a688c8a7d7a6c6346a"
server_key_iv: "f693f066548a6398b2052503cd63d044"
server_key_version: "encryptionkey@0"
I20260618 22:31:20.580873 6595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:20.593148 6595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:20.596170 6595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:20.596328 6595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:20.603236 6595 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:33211
I20260618 22:31:20.603253 6662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:33211 every 8 connection(s)
I20260618 22:31:20.604183 6595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:20.606915 6663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:20.611596 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6595
I20260618 22:31:20.611734 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:20.611950 17358 external_mini_cluster.cc:1428] Setting key fad1023077892d8ca2e28dfd8cec1e40
I20260618 22:31:20.612489 6663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc: Bootstrap starting.
I20260618 22:31:20.614843 6663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:20.615651 6663 log.cc:824] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:20.617697 6663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc: No bootstrap required, opened a new log
I20260618 22:31:20.620867 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:20.613199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51875 (local address 127.16.243.190:33211)
0618 22:31:20.613593 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:20.613608 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:20.613645 (+ 37us) server_negotiation.cc:407] Connection header received
0618 22:31:20.614280 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:20.614297 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:20.614696 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:20.615019 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:20.615349 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:20.616532 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:20.617702 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:20.618220 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:20.618350 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:20.619218 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:20.619223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:20.619235 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:20.619633 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:20.619686 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:20.619708 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:20.619804 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:20.619981 (+ 177us) server_negotiation.cc:299] Negotiation successful
0618 22:31:20.620114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260618 22:31:20.621542 6663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } }
I20260618 22:31:20.621752 6663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:20.621791 6663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b11894310694bad95c9a0af4c8954bc, State: Initialized, Role: FOLLOWER
I20260618 22:31:20.622197 6663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } }
I20260618 22:31:20.622313 6663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:20.622370 6663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:20.622457 6663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:20.623425 6663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } }
I20260618 22:31:20.623602 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:20.623775 6663 leader_election.cc:302] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b11894310694bad95c9a0af4c8954bc; no voters:
I20260618 22:31:20.624110 6663 leader_election.cc:288] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:20.624179 6668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:20.624379 6668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [term 1 LEADER]: Becoming Leader. State: Replica: 6b11894310694bad95c9a0af4c8954bc, State: Running, Role: LEADER
I20260618 22:31:20.624691 6668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } }
I20260618 22:31:20.625409 6663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:20.625855 6670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b11894310694bad95c9a0af4c8954bc. Latest consensus state: current_term: 1 leader_uuid: "6b11894310694bad95c9a0af4c8954bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } } }
I20260618 22:31:20.626003 6670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:20.626205 6669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b11894310694bad95c9a0af4c8954bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b11894310694bad95c9a0af4c8954bc" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 33211 } } }
I20260618 22:31:20.626296 6669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:20.626901 6676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:20.629988 6676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:20.631193 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:20.625751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38663 (local address 127.16.243.190:33211)
0618 22:31:20.626441 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:20.626445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:20.626826 (+ 381us) server_negotiation.cc:407] Connection header received
0618 22:31:20.627013 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:20.627017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:20.627067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:20.627119 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:20.627564 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:20.628362 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:20.629522 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:20.629758 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:20.629845 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:20.630718 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:20.630722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:20.630725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:20.630739 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:20.630769 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:20.630783 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:20.630839 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:20.631031 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 22:31:20.631065 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608}
I20260618 22:31:20.636881 6676 catalog_manager.cc:1347] Generated new cluster ID: f0ae94ad5f2c43ae8fb5a552c439e375
I20260618 22:31:20.636965 6676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:20.651357 6676 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:20.652653 6676 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:20.662778 6676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b11894310694bad95c9a0af4c8954bc: Generated new TSK 0
I20260618 22:31:20.663686 6676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:20.668597 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6595
W20260618 22:31:20.673343 6580 connection.cc:535] client connection to 127.16.243.190:33211 recv error: Network error: failed to read from TLS socket (remote: 127.16.243.190:33211): Connection reset by peer (error 104)
2026-06-18T22:31:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:20 dist-test-slave-jm5g krb5kdc[6694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:20 dist-test-slave-jm5g krb5kdc[6694](info): set up 2 sockets
Jun 18 22:31:20 dist-test-slave-jm5g krb5kdc[6694](info): commencing operation
krb5kdc: starting...
W20260618 22:31:21.626627 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:21 dist-test-slave-jm5g krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:21.737484 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37663
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:21.846055 6707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:21.846335 6707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:21.846397 6707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:21.850665 6707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:21.850734 6707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:21.850760 6707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:21.850781 6707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:21.850811 6707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:21.856237 6707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:37663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:21.857318 6707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:21.858332 6707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:21.866343 6713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:21.866403 6712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:21.866498 6707 server_base.cc:1034] running on GCE node
W20260618 22:31:21.866343 6715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:21.867220 6707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:21.868372 6707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:21.869540 6707 hybrid_clock.cc:648] HybridClock initialized: now 1781821881869523 us; error 29 us; skew 500 ppm
Jun 18 22:31:21 dist-test-slave-jm5g krb5kdc[6694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821881, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:21.872296 6707 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:21.873238 6707 webserver.cc:460] Webserver started at http://127.16.243.190:39217/ using document root <none> and password file <none>
I20260618 22:31:21.873780 6707 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:21.873844 6707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:21.874032 6707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:21.876102 6707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "5a161c396cf346988e908434b95d4859"
format_stamp: "Formatted at 2026-06-18 22:31:21 on dist-test-slave-jm5g"
server_key: "5f75e1ff70977a6c9ce63cb14dfb9094"
server_key_iv: "cfc1666e075d276846ec726cb4795929"
server_key_version: "encryptionkey@0"
I20260618 22:31:21.876641 6707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "5a161c396cf346988e908434b95d4859"
format_stamp: "Formatted at 2026-06-18 22:31:21 on dist-test-slave-jm5g"
server_key: "5f75e1ff70977a6c9ce63cb14dfb9094"
server_key_iv: "cfc1666e075d276846ec726cb4795929"
server_key_version: "encryptionkey@0"
I20260618 22:31:21.880353 6707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:31:21.882761 6722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:21.883966 6707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:21.884094 6707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "5a161c396cf346988e908434b95d4859"
format_stamp: "Formatted at 2026-06-18 22:31:21 on dist-test-slave-jm5g"
server_key: "5f75e1ff70977a6c9ce63cb14dfb9094"
server_key_iv: "cfc1666e075d276846ec726cb4795929"
server_key_version: "encryptionkey@0"
I20260618 22:31:21.884194 6707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:21.888795 6707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:21.891690 6707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:21.891856 6707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:21.898288 6707 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:37663
I20260618 22:31:21.898322 6774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:37663 every 8 connection(s)
I20260618 22:31:21.899245 6707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:21.901975 6775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:21.903190 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6707
I20260618 22:31:21.903311 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:21.903569 17358 external_mini_cluster.cc:1428] Setting key 755fcbd55abd5046b6cc169b67d1babe
I20260618 22:31:21.908594 6775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: Bootstrap starting.
I20260618 22:31:21.909149 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:21.904943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38923 (local address 127.16.243.190:37663)
0618 22:31:21.905398 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:21.905417 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:21.905461 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:31:21.906265 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:21.906291 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:21.906711 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:21.907095 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:21.907722 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:21.907727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:21.907743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:21.907752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:21.907812 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:21.907838 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:21.907986 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:21.908062 (+ 76us) server_negotiation.cc:299] Negotiation successful
0618 22:31:21.908219 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260618 22:31:21.911247 6775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:21.912045 6775 log.cc:824] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:21.912096 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:21.914165 6775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: No bootstrap required, opened a new log
I20260618 22:31:21.915663 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:21.914109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60423 (local address 127.16.243.190:37663)
0618 22:31:21.914210 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:21.914215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:21.914403 (+ 188us) server_negotiation.cc:407] Connection header received
0618 22:31:21.914512 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:21.914515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:21.914585 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:21.914648 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:21.915322 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:21.915325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:21.915328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:21.915339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:21.915366 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:21.915376 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:21.915445 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:21.915513 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 22:31:21.915551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:31:21.918027 6775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } }
I20260618 22:31:21.918257 6775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:21.918305 6775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a161c396cf346988e908434b95d4859, State: Initialized, Role: FOLLOWER
I20260618 22:31:21.918781 6775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } }
I20260618 22:31:21.918949 6775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:21.919028 6775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:21.919129 6775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:21.920116 6775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } }
I20260618 22:31:21.920451 6775 leader_election.cc:302] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a161c396cf346988e908434b95d4859; no voters:
I20260618 22:31:21.920771 6775 leader_election.cc:288] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:21.920863 6785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:21.921068 6785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [term 1 LEADER]: Becoming Leader. State: Replica: 5a161c396cf346988e908434b95d4859, State: Running, Role: LEADER
I20260618 22:31:21.921422 6785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } }
I20260618 22:31:21.921970 6775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:21.923264 6787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a161c396cf346988e908434b95d4859. Latest consensus state: current_term: 1 leader_uuid: "5a161c396cf346988e908434b95d4859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } } }
I20260618 22:31:21.923278 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a161c396cf346988e908434b95d4859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a161c396cf346988e908434b95d4859" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37663 } } }
I20260618 22:31:21.923391 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:21.923390 6787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859 [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:21.926527 6800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:21.926642 6800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:21.926703 6795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:21.927374 6795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:21.932709 6795 catalog_manager.cc:1347] Generated new cluster ID: db8d171e24db482d80e252f132b173e3
I20260618 22:31:21.932783 6795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:21.953974 6795 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:21.954296 6795 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:21.967713 6795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a161c396cf346988e908434b95d4859: Generated new TSK 0
I20260618 22:31:21.968451 6795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:21.976883 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6707
2026-06-18T22:31:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:22 dist-test-slave-jm5g krb5kdc[6806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:22 dist-test-slave-jm5g krb5kdc[6806](info): set up 2 sockets
Jun 18 22:31:22 dist-test-slave-jm5g krb5kdc[6806](info): commencing operation
krb5kdc: starting...
W20260618 22:31:22.930593 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:23.043165 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35801
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:23.148875 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:23.149139 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:23.149195 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:23.153136 6819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:23.153195 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:23.153221 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:23.153241 6819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:23.153262 6819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:23.158464 6819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:23.159526 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:23.160522 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:23.168584 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:23.168609 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:23.168630 6819 server_base.cc:1034] running on GCE node
W20260618 22:31:23.168584 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:23.169268 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:23.170339 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:23.171481 6819 hybrid_clock.cc:648] HybridClock initialized: now 1781821883171448 us; error 46 us; skew 500 ppm
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821883, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:23.173936 6819 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:23.174808 6819 webserver.cc:460] Webserver started at http://127.16.243.190:33373/ using document root <none> and password file <none>
I20260618 22:31:23.175333 6819 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:23.175396 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:23.175576 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:23.177526 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "95db48b01e794fc093d78806c1503fbb"
format_stamp: "Formatted at 2026-06-18 22:31:23 on dist-test-slave-jm5g"
server_key: "89b905d9d2012b8ce70f94f3a952fb65"
server_key_iv: "9d7643f6118de00970dccbecaf2292ae"
server_key_version: "encryptionkey@0"
I20260618 22:31:23.178054 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "95db48b01e794fc093d78806c1503fbb"
format_stamp: "Formatted at 2026-06-18 22:31:23 on dist-test-slave-jm5g"
server_key: "89b905d9d2012b8ce70f94f3a952fb65"
server_key_iv: "9d7643f6118de00970dccbecaf2292ae"
server_key_version: "encryptionkey@0"
I20260618 22:31:23.181581 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:31:23.183914 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:23.185114 6819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:31:23.185222 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "95db48b01e794fc093d78806c1503fbb"
format_stamp: "Formatted at 2026-06-18 22:31:23 on dist-test-slave-jm5g"
server_key: "89b905d9d2012b8ce70f94f3a952fb65"
server_key_iv: "9d7643f6118de00970dccbecaf2292ae"
server_key_version: "encryptionkey@0"
I20260618 22:31:23.185313 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:23.203207 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:23.206152 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:23.206331 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:23.212818 6819 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:35801
I20260618 22:31:23.212828 6886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:35801 every 8 connection(s)
I20260618 22:31:23.213744 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:23.216471 6887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:23.218544 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6819
I20260618 22:31:23.218712 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:23.218999 17358 external_mini_cluster.cc:1428] Setting key a3932ff3f82b01a6cd25bed98378d14f
I20260618 22:31:23.222628 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb: Bootstrap starting.
I20260618 22:31:23.224828 6887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:23.225530 6887 log.cc:824] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:23.227829 6887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb: No bootstrap required, opened a new log
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:23.231990 6887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } }
I20260618 22:31:23.232278 6887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:23.232340 6887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95db48b01e794fc093d78806c1503fbb, State: Initialized, Role: FOLLOWER
I20260618 22:31:23.232764 6887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } }
I20260618 22:31:23.232890 6887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:23.232952 6887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:23.233050 6887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:23.233198 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:23.220428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54499 (local address 127.16.243.190:35801)
0618 22:31:23.220853 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:23.220868 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:23.220904 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:23.221713 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:23.221733 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:23.222104 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:23.222411 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:23.223268 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:23.224266 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:23.225121 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:23.225607 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:23.225777 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:23.228561 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:23.228566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:23.228579 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:23.228605 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:23.230122 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:23.230615 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:23.230619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:23.230623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:23.230727 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:23.231007 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:23.231009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:23.231011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:23.231470 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:23.231697 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:23.231978 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:31:23.232293 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":124,"threads_started":1}
I20260618 22:31:23.233970 6887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } }
I20260618 22:31:23.234331 6887 leader_election.cc:302] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95db48b01e794fc093d78806c1503fbb; no voters:
I20260618 22:31:23.234702 6887 leader_election.cc:288] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:23.234779 6892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:23.234993 6892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [term 1 LEADER]: Becoming Leader. State: Replica: 95db48b01e794fc093d78806c1503fbb, State: Running, Role: LEADER
I20260618 22:31:23.235338 6892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } }
I20260618 22:31:23.235925 6887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:23.236267 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:23.236518 6892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95db48b01e794fc093d78806c1503fbb. Latest consensus state: current_term: 1 leader_uuid: "95db48b01e794fc093d78806c1503fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } } }
I20260618 22:31:23.236689 6892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:23.236917 6893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95db48b01e794fc093d78806c1503fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db48b01e794fc093d78806c1503fbb" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 35801 } } }
I20260618 22:31:23.237025 6893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:23.237164 6901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:23.240418 6901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:23.245718 6901 catalog_manager.cc:1347] Generated new cluster ID: 7194aaa98ff14270b77924f2c34f8863
I20260618 22:31:23.245834 6901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:23.247313 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:23.238360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38549 (local address 127.16.243.190:35801)
0618 22:31:23.238459 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:23.238463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:23.238717 (+ 254us) server_negotiation.cc:407] Connection header received
0618 22:31:23.238858 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:23.238861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:23.238908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:23.238979 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:23.239759 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:23.240609 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:23.241876 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:23.242200 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:23.242302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:23.243944 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:23.243949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:23.243952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:23.243979 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:23.245564 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:23.246089 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:23.246092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:23.246094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:23.246185 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:23.246470 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:23.246473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:23.246475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:23.246702 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:23.246814 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:23.247062 (+ 248us) server_negotiation.cc:299] Negotiation successful
0618 22:31:23.247174 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 22:31:23.260397 6901 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:23.261318 6901 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:23.269098 6901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95db48b01e794fc093d78806c1503fbb: Generated new TSK 0
I20260618 22:31:23.269794 6901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:23.276475 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6819
2026-06-18T22:31:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6918](info): set up 2 sockets
Jun 18 22:31:23 dist-test-slave-jm5g krb5kdc[6918](info): commencing operation
krb5kdc: starting...
W20260618 22:31:24.234516 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:24 dist-test-slave-jm5g krb5kdc[6918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:24.347455 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44183
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:44183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:24.454231 6931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:24.454509 6931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:24.454578 6931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:24.458614 6931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:24.458685 6931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:24.458703 6931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:24.458719 6931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:24.458732 6931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:24.464057 6931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:44183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:44183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:24.465119 6931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:24.466118 6931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:24.474447 6937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:24.474515 6931 server_base.cc:1034] running on GCE node
W20260618 22:31:24.474458 6939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:24.474472 6936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:24.475353 6931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:24.476476 6931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:24.477629 6931 hybrid_clock.cc:648] HybridClock initialized: now 1781821884477604 us; error 40 us; skew 500 ppm
Jun 18 22:31:24 dist-test-slave-jm5g krb5kdc[6918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821884, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:24.480289 6931 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:24.481199 6931 webserver.cc:460] Webserver started at http://127.16.243.190:37649/ using document root <none> and password file <none>
I20260618 22:31:24.481735 6931 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:24.481798 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:24.482002 6931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:24.484118 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "842c947a1b254a82af51479ae926efb6"
format_stamp: "Formatted at 2026-06-18 22:31:24 on dist-test-slave-jm5g"
server_key: "c8160636f449624d61be04ac71265689"
server_key_iv: "a298f0bd8596da05cfb98802b1790a07"
server_key_version: "encryptionkey@0"
I20260618 22:31:24.484649 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "842c947a1b254a82af51479ae926efb6"
format_stamp: "Formatted at 2026-06-18 22:31:24 on dist-test-slave-jm5g"
server_key: "c8160636f449624d61be04ac71265689"
server_key_iv: "a298f0bd8596da05cfb98802b1790a07"
server_key_version: "encryptionkey@0"
I20260618 22:31:24.488320 6931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:31:24.490708 6946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:24.492012 6931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:24.492138 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "842c947a1b254a82af51479ae926efb6"
format_stamp: "Formatted at 2026-06-18 22:31:24 on dist-test-slave-jm5g"
server_key: "c8160636f449624d61be04ac71265689"
server_key_iv: "a298f0bd8596da05cfb98802b1790a07"
server_key_version: "encryptionkey@0"
I20260618 22:31:24.492241 6931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:24.508880 6931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:24.511775 6931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:24.511968 6931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:24.518726 6931 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:44183
I20260618 22:31:24.518739 6998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:44183 every 8 connection(s)
I20260618 22:31:24.519657 6931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:24.522372 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:24.523120 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 6931
I20260618 22:31:24.523269 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:24.523535 17358 external_mini_cluster.cc:1428] Setting key e23c2c1cde6348674b942e865b0c7ca3
I20260618 22:31:24.529752 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6: Bootstrap starting.
I20260618 22:31:24.532815 6999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:24.533790 6999 log.cc:824] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:24.535408 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:24.524927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33019 (local address 127.16.243.190:44183)
0618 22:31:24.525343 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:24.525363 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:24.525658 (+ 295us) server_negotiation.cc:407] Connection header received
0618 22:31:24.526619 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:24.526645 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:24.527137 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:24.527516 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:24.528323 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:24.529786 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:24.531028 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:24.531723 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:24.531865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:24.532977 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:24.532983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:24.533002 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:24.533508 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:24.533579 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:24.533608 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:24.533691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:24.534180 (+ 489us) server_negotiation.cc:299] Negotiation successful
0618 22:31:24.534403 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260618 22:31:24.538214 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6: No bootstrap required, opened a new log
I20260618 22:31:24.540236 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:24.542941 6999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } }
I20260618 22:31:24.543303 6999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:24.543360 6999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842c947a1b254a82af51479ae926efb6, State: Initialized, Role: FOLLOWER
I20260618 22:31:24.543906 6999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } }
I20260618 22:31:24.544075 6999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:24.544157 6999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:24.544262 6999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:24.545481 6999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } }
I20260618 22:31:24.545838 6999 leader_election.cc:302] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842c947a1b254a82af51479ae926efb6; no voters:
I20260618 22:31:24.546150 6999 leader_election.cc:288] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:24.546236 7009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:24.546401 7009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [term 1 LEADER]: Becoming Leader. State: Replica: 842c947a1b254a82af51479ae926efb6, State: Running, Role: LEADER
I20260618 22:31:24.546737 7009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } }
I20260618 22:31:24.547520 6999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:24.547640 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:24.542496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46479 (local address 127.16.243.190:44183)
0618 22:31:24.542702 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:24.542706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:24.542835 (+ 129us) server_negotiation.cc:407] Connection header received
0618 22:31:24.542989 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:24.542992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:24.543039 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:24.543124 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:24.543831 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:24.544612 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:24.545662 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:24.545946 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:24.546049 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:24.547189 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:24.547192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:24.547195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:24.547208 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:24.547235 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:24.547247 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:24.547304 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:24.547392 (+ 88us) server_negotiation.cc:299] Negotiation successful
0618 22:31:24.547446 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260618 22:31:24.548507 7010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842c947a1b254a82af51479ae926efb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } } }
I20260618 22:31:24.548712 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842c947a1b254a82af51479ae926efb6. Latest consensus state: current_term: 1 leader_uuid: "842c947a1b254a82af51479ae926efb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c947a1b254a82af51479ae926efb6" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 44183 } } }
I20260618 22:31:24.548810 7010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:24.548842 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:24.549222 7018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:24.552186 7018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:24.557943 7018 catalog_manager.cc:1347] Generated new cluster ID: ccba2b8cf72640509c317295aaa2a811
I20260618 22:31:24.558025 7018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:24.567771 7018 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:24.569072 7018 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:24.581645 7018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842c947a1b254a82af51479ae926efb6: Generated new TSK 0
I20260618 22:31:24.582401 7018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:24.592603 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 6931
2026-06-18T22:31:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:24 dist-test-slave-jm5g krb5kdc[7030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:24 dist-test-slave-jm5g krb5kdc[7030](info): set up 2 sockets
Jun 18 22:31:24 dist-test-slave-jm5g krb5kdc[7030](info): commencing operation
krb5kdc: starting...
W20260618 22:31:25.534705 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:25 dist-test-slave-jm5g krb5kdc[7030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:25.645417 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43419
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:25.753170 7043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:25.753453 7043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:25.753494 7043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:25.757467 7043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:25.757530 7043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:25.757550 7043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:25.757565 7043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:25.757578 7043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:25.762923 7043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:25.763969 7043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:25.764948 7043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:25.773303 7051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:25.773303 7049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:25.773340 7043 server_base.cc:1034] running on GCE node
W20260618 22:31:25.773363 7048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:25.774135 7043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:25.775305 7043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:25.776481 7043 hybrid_clock.cc:648] HybridClock initialized: now 1781821885776460 us; error 36 us; skew 500 ppm
Jun 18 22:31:25 dist-test-slave-jm5g krb5kdc[7030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821885, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:25.782213 7043 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:25.783300 7043 webserver.cc:460] Webserver started at http://127.16.243.190:43687/ using document root <none> and password file <none>
I20260618 22:31:25.783883 7043 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:25.783957 7043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:25.784171 7043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:25.786293 7043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "626ae322ac5d436585ef212a82c0234b"
format_stamp: "Formatted at 2026-06-18 22:31:25 on dist-test-slave-jm5g"
server_key: "b76f825de30bb76111a996dcfbe4f693"
server_key_iv: "de13afea8ea48dda431e5a74d797e2f5"
server_key_version: "encryptionkey@0"
I20260618 22:31:25.786921 7043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "626ae322ac5d436585ef212a82c0234b"
format_stamp: "Formatted at 2026-06-18 22:31:25 on dist-test-slave-jm5g"
server_key: "b76f825de30bb76111a996dcfbe4f693"
server_key_iv: "de13afea8ea48dda431e5a74d797e2f5"
server_key_version: "encryptionkey@0"
I20260618 22:31:25.791103 7043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:31:25.793646 7058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:25.794929 7043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:25.795043 7043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "626ae322ac5d436585ef212a82c0234b"
format_stamp: "Formatted at 2026-06-18 22:31:25 on dist-test-slave-jm5g"
server_key: "b76f825de30bb76111a996dcfbe4f693"
server_key_iv: "de13afea8ea48dda431e5a74d797e2f5"
server_key_version: "encryptionkey@0"
I20260618 22:31:25.795145 7043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:25.799636 7043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:25.802740 7043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:25.802911 7043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:25.809413 7043 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:43419
I20260618 22:31:25.809469 7110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:43419 every 8 connection(s)
I20260618 22:31:25.810339 7043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:25.811201 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7043
I20260618 22:31:25.811311 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:25.811545 17358 external_mini_cluster.cc:1428] Setting key 9d45a877c9219d4b3b83bcf6d1cedcb9
I20260618 22:31:25.813498 7111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:25.816550 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:25.812871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39071 (local address 127.16.243.190:43419)
0618 22:31:25.813327 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:25.813342 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:25.813378 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:25.814061 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:25.814078 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:25.814361 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:25.814690 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:25.815433 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:25.815437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:25.815448 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:25.815458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:25.815506 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:25.815524 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:25.815623 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:25.815710 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 22:31:25.815847 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":138,"threads_started":1}
I20260618 22:31:25.819293 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:25.819763 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b: Bootstrap starting.
I20260618 22:31:25.822387 7111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:25.823227 7111 log.cc:824] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:25.823402 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:25.821601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34673 (local address 127.16.243.190:43419)
0618 22:31:25.821728 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:25.821733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:25.821970 (+ 237us) server_negotiation.cc:407] Connection header received
0618 22:31:25.822071 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:25.822074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:25.822113 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:25.822194 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:25.822965 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:25.822968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:25.822971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:31:25.822983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:25.823015 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:31:25.823028 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:25.823135 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:25.823252 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 22:31:25.823292 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:31:25.825214 7111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b: No bootstrap required, opened a new log
I20260618 22:31:25.828706 7111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } }
I20260618 22:31:25.828943 7111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:31:25.828984 7111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626ae322ac5d436585ef212a82c0234b, State: Initialized, Role: FOLLOWER
I20260618 22:31:25.829348 7111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } }
I20260618 22:31:25.829465 7111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:31:25.829509 7111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:31:25.829576 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:25.830439 7111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } }
I20260618 22:31:25.830776 7111 leader_election.cc:302] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626ae322ac5d436585ef212a82c0234b; no voters:
I20260618 22:31:25.831060 7111 leader_election.cc:288] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:31:25.831187 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:25.831382 7121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [term 1 LEADER]: Becoming Leader. State: Replica: 626ae322ac5d436585ef212a82c0234b, State: Running, Role: LEADER
I20260618 22:31:25.831707 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } }
I20260618 22:31:25.832175 7111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:25.833210 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626ae322ac5d436585ef212a82c0234b. Latest consensus state: current_term: 1 leader_uuid: "626ae322ac5d436585ef212a82c0234b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } } }
I20260618 22:31:25.833338 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:25.833617 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626ae322ac5d436585ef212a82c0234b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626ae322ac5d436585ef212a82c0234b" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 43419 } } }
I20260618 22:31:25.833700 7130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:25.833783 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:25.836881 7130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:25.842383 7130 catalog_manager.cc:1347] Generated new cluster ID: b8eb6a6d2358463aa503625c15f99379
I20260618 22:31:25.842458 7130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:25.861836 7130 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:25.862160 7130 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:25.872514 7130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626ae322ac5d436585ef212a82c0234b: Generated new TSK 0
I20260618 22:31:25.873238 7130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:31:25.886062 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7043
2026-06-18T22:31:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (23498 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:25 dist-test-slave-jm5g krb5kdc[7142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:25 dist-test-slave-jm5g krb5kdc[7142](info): set up 2 sockets
Jun 18 22:31:25 dist-test-slave-jm5g krb5kdc[7142](info): commencing operation
krb5kdc: starting...
W20260618 22:31:26.834666 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:26 dist-test-slave-jm5g krb5kdc[7142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:26.946131 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37163
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:27.056075 7155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:27.056394 7155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:27.056429 7155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:27.056493 7155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:27.060575 7155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:27.060653 7155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:27.060679 7155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:27.060700 7155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:27.060729 7155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:27.066161 7155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:27.067322 7155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:27.068341 7155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:27.076713 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:27.076745 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:27.076730 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:27.076925 7155 server_base.cc:1034] running on GCE node
I20260618 22:31:27.077457 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:27.078577 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:27.079743 7155 hybrid_clock.cc:648] HybridClock initialized: now 1781821887079723 us; error 34 us; skew 500 ppm
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:27.082424 7155 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:27.083321 7155 webserver.cc:460] Webserver started at http://127.16.243.190:34753/ using document root <none> and password file <none>
I20260618 22:31:27.083847 7155 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:27.083904 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:27.084069 7155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:27.086055 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "558524616c5b4f788c1c28006d2a141c"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "05cc0985ef39cc94f349e3cd7ceb815b"
server_key_iv: "20440b87db91096692ab23abab729062"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.086616 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "558524616c5b4f788c1c28006d2a141c"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "05cc0985ef39cc94f349e3cd7ceb815b"
server_key_iv: "20440b87db91096692ab23abab729062"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.090197 7155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:31:27.092700 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:27.093927 7155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:27.094031 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "558524616c5b4f788c1c28006d2a141c"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "05cc0985ef39cc94f349e3cd7ceb815b"
server_key_iv: "20440b87db91096692ab23abab729062"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.094123 7155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:27.120853 7155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:27.124079 7155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:27.124238 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:27.130797 7155 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:37163
I20260618 22:31:27.131089 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:37163 every 8 connection(s)
I20260618 22:31:27.131871 7155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:27.132076 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7155
I20260618 22:31:27.132205 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:27.132498 17358 external_mini_cluster.cc:1428] Setting key 2fe623afc513e6bed963c9e756c1ab71
I20260618 22:31:27.134754 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:27.138398 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.189@KRBTEST.COM" created.
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:27.153491 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.141562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.16.243.190:37163)
0618 22:31:27.141815 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.141823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.141859 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:27.142617 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.142622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.142812 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.142993 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.143827 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.144680 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.146025 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.146461 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.146629 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.149241 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.149247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.149258 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.149284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.150803 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.151233 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.151239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.151241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.151353 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.151594 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.151596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.151598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.151983 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.152093 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.152447 (+ 354us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.152638 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260618 22:31:27.153497 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.140680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37163 (local address 127.0.0.1:56046)
0618 22:31:27.141193 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.141242 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.142281 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.142993 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.142999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.143444 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.143689 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.143702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.144793 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.144796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.145895 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.145906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.146645 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.146649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.146778 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.147356 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.147379 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.149092 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.150907 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.150916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.150925 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.151138 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.151422 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.151424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.151425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.151530 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.152084 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.152090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.152216 (+ 126us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.152367 (+ 151us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.152721 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":76,"thread_start_us":126,"threads_started":1}
I20260618 22:31:27.155916 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:27.156621 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.156361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51384)
0618 22:31:27.156474 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.156547 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:31:27.156798 7172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
I20260618 22:31:27.157639 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.157412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51386)
0618 22:31:27.157525 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.157574 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:31:27.157780 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:45563: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
I20260618 22:31:27.158396 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:45563
--webserver_interface=127.16.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:31:27.207024 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } attempt: 1
I20260618 22:31:27.207875 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.207589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51388)
0618 22:31:27.207736 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.207798 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260618 22:31:27.208539 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.208334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51400)
0618 22:31:27.208431 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.208481 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:31:27.208688 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:45563: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
W20260618 22:31:27.266099 7230 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:27.266402 7230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:27.266440 7230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:27.266513 7230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:27.270520 7230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:27.270614 7230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:27.270641 7230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:27.270661 7230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:27.270684 7230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:27.276269 7230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:27.276845 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } attempt: 2
I20260618 22:31:27.277498 7230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:27.277673 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.277352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51414)
0618 22:31:27.277507 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.277572 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:31:27.278342 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.278126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51428)
0618 22:31:27.278239 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.278284 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 22:31:27.278452 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:45563: Network error: Client connection negotiation failed: client connection to 127.16.243.189:45563: connect: Connection refused (error 111)
I20260618 22:31:27.278644 7230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:27.286755 7230 server_base.cc:1034] running on GCE node
W20260618 22:31:27.286746 7236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:27.286741 7238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:27.286912 7235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:27.287509 7230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:27.288560 7230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:27.289712 7230 hybrid_clock.cc:648] HybridClock initialized: now 1781821887289695 us; error 30 us; skew 500 ppm
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:27.292356 7230 init.cc:377] Logged in from keytab as kudu/127.16.243.189@KRBTEST.COM (short username kudu)
I20260618 22:31:27.293246 7230 webserver.cc:460] Webserver started at http://127.16.243.189:36341/ using document root <none> and password file <none>
I20260618 22:31:27.293776 7230 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:27.293838 7230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:27.294023 7230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:27.296054 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data/instance:
uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "3cc0c413533eda8dd6919db4adbf69a3"
server_key_iv: "ebdbc6949af49f2a543aa10f1f131489"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.296583 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal/instance:
uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "3cc0c413533eda8dd6919db4adbf69a3"
server_key_iv: "ebdbc6949af49f2a543aa10f1f131489"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.300279 7230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:31:27.302665 7245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:27.303966 7230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:31:27.304078 7230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "3cc0c413533eda8dd6919db4adbf69a3"
server_key_iv: "ebdbc6949af49f2a543aa10f1f131489"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.304181 7230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:27.330890 7230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:27.334093 7230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:27.334266 7230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:27.340991 7230 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.189:45563
I20260618 22:31:27.341253 7297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.189:45563 every 8 connection(s)
I20260618 22:31:27.341918 7230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
I20260618 22:31:27.344084 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7230
I20260618 22:31:27.344198 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-1/wal/instance
I20260618 22:31:27.344439 17358 external_mini_cluster.cc:1428] Setting key 16eaee397914f0a7fcbbb79e87954389
I20260618 22:31:27.344702 7298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:27.349296 7298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.188@KRBTEST.COM" created.
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:27.363226 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.351569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.16.243.190:37163)
0618 22:31:27.351717 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.351721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.352541 (+ 820us) server_negotiation.cc:407] Connection header received
0618 22:31:27.353527 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.353530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.353577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.353659 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.354960 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.355486 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.356591 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.356844 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.356935 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.359597 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.359600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.359602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.359629 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.360983 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.361604 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.361606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.361608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.361676 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.361950 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.361952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.361953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.362203 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.362279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.362876 (+ 597us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.363045 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:31:27.364035 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.351900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37163 (local address 127.0.0.1:56052)
0618 22:31:27.352379 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.352431 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.353337 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.353874 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.353883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.354259 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.354814 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.354827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.355618 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.355621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.356467 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.356476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.356987 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.356991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.357154 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.357704 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.357720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.359409 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.361107 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.361114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.361131 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.361516 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.361743 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.361746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.361747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.361883 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.362288 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.362293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.362606 (+ 313us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.362792 (+ 186us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.363153 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260618 22:31:27.365384 7298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
I20260618 22:31:27.369256 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:45359
--webserver_interface=127.16.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:31:27.369593 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } attempt: 3
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:31:27.375754 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.365831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51436)
0618 22:31:27.365946 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.365959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.366030 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.366611 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.366615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.366748 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.366838 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.366843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.367803 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.367805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.368429 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.368439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.369160 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.369163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.369237 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.369693 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.369707 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.371187 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.374053 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.374056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.374058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.374303 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.374639 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.374641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.374643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.374717 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.375125 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.375128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.375209 (+ 81us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.375326 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.375579 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 22:31:27.375792 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.365946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.16.243.189:45563)
0618 22:31:27.366249 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.366260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.366278 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:31:27.366313 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.366317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.366471 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.366620 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.366926 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.367700 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.368530 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.369031 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.369156 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.371329 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.371339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.371356 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.371380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.373931 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.374423 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.374430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.374432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.374519 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.374797 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.374800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.374801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.375025 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.375143 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.375412 (+ 269us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.375653 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":213,"thread_start_us":119,"threads_started":1}
I20260618 22:31:27.377959 7298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:27.378777 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.378468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48036)
0618 22:31:27.378624 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.378700 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260618 22:31:27.379055 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.370023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51442)
0618 22:31:27.370149 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.370164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.370240 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.370760 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.370763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.370941 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.371033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.371039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.371911 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.371915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.372446 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.372453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.373556 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.373560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.373677 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.374126 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.374138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.375571 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.377609 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.377612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.377614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.377849 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.378153 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.378156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.378158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.378244 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.378644 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.378647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.378744 (+ 97us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.378826 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.378947 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:31:27.379067 7248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.379518 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.16.243.189:45563)
0618 22:31:27.370491 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.370500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.370513 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:31:27.370545 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.370569 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.370610 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.370725 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.371141 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.371825 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.372551 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.373429 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.373533 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.375740 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.375743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.375746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.375771 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.377491 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.377955 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.377958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.377960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.378054 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.378348 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.378350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.378351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.378502 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.379211 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.379279 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.379371 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
I20260618 22:31:27.380072 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:27.380242 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.380032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48042)
0618 22:31:27.380136 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.380187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260618 22:31:27.380717 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.380509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48056)
0618 22:31:27.380618 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.380662 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:31:27.381247 7298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.381592 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.381382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48064)
0618 22:31:27.381496 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.381536 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260618 22:31:27.381698 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.412832 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 1
I20260618 22:31:27.413661 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.413365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48076)
0618 22:31:27.413520 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.413580 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 22:31:27.414302 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.414109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48082)
0618 22:31:27.414207 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.414248 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:31:27.414415 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.430526 7298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 1
I20260618 22:31:27.431403 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.431120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48088)
0618 22:31:27.431264 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.431325 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 22:31:27.432045 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.431848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48098)
0618 22:31:27.431946 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.431989 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:31:27.432161 7298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
W20260618 22:31:27.480809 7305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:27.481117 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:27.481168 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:27.481231 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:27.485306 7305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:27.485373 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:27.485401 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:27.485425 7305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:27.485455 7305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:27.489560 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 2
I20260618 22:31:27.490376 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.490083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48114)
0618 22:31:27.490233 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.490296 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:31:27.491042 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.490840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48122)
0618 22:31:27.490943 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.490987 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 22:31:27.491074 7305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:37163,127.16.243.189:45563,127.16.243.188:45359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
W20260618 22:31:27.491161 7223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.492168 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:27.493171 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:27.500492 7298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 2
I20260618 22:31:27.501312 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.500985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48128)
0618 22:31:27.501154 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.501217 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260618 22:31:27.501685 7311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:27.501736 7305 server_base.cc:1034] running on GCE node
W20260618 22:31:27.501705 7312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:27.501945 7314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:27.502274 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.502050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48142)
0618 22:31:27.502164 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.502215 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 22:31:27.502413 7298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:45359: Network error: Client connection negotiation failed: client connection to 127.16.243.188:45359: connect: Connection refused (error 111)
I20260618 22:31:27.502641 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:27.503803 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:27.504944 7305 hybrid_clock.cc:648] HybridClock initialized: now 1781821887504914 us; error 42 us; skew 500 ppm
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:27.507560 7305 init.cc:377] Logged in from keytab as kudu/127.16.243.188@KRBTEST.COM (short username kudu)
I20260618 22:31:27.508383 7305 webserver.cc:460] Webserver started at http://127.16.243.188:37255/ using document root <none> and password file <none>
I20260618 22:31:27.508935 7305 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:27.509001 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:27.509184 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:27.511267 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data/instance:
uuid: "699a27a30f63419998ba0caf411355f9"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "d39e81cd4137c022adf8e183340c3672"
server_key_iv: "be6f801641da384c829beab86dfba82b"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.511796 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal/instance:
uuid: "699a27a30f63419998ba0caf411355f9"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "d39e81cd4137c022adf8e183340c3672"
server_key_iv: "be6f801641da384c829beab86dfba82b"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.515426 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:31:27.517763 7321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:27.518996 7305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:27.519106 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "699a27a30f63419998ba0caf411355f9"
format_stamp: "Formatted at 2026-06-18 22:31:27 on dist-test-slave-jm5g"
server_key: "d39e81cd4137c022adf8e183340c3672"
server_key_iv: "be6f801641da384c829beab86dfba82b"
server_key_version: "encryptionkey@0"
I20260618 22:31:27.519200 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:27.530957 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:27.533744 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:27.533924 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:27.540616 7305 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.188:45359
I20260618 22:31:27.540867 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.188:45359 every 8 connection(s)
I20260618 22:31:27.541590 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
I20260618 22:31:27.544486 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:27.545315 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7305
I20260618 22:31:27.545424 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821616719733-17358-0/minicluster-data/master-2/wal/instance
I20260618 22:31:27.545670 17358 external_mini_cluster.cc:1428] Setting key f9b4abe76b1dea0887d2cba91e261c58
I20260618 22:31:27.548380 7374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:27.559283 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.550295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.16.243.190:37163)
0618 22:31:27.550410 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.550414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.550757 (+ 343us) server_negotiation.cc:407] Connection header received
0618 22:31:27.550874 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.550877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.550924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.551000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.551650 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.552202 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.553129 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.553460 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.553572 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.556298 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.556301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.556303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.556326 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.557706 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.558045 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.558048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.558049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.558118 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.558367 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.558369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.558370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.558538 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.558653 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.559029 (+ 376us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.559147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:31:27.560220 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.550242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:37163 (local address 127.0.0.1:56062)
0618 22:31:27.550640 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.550680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.550793 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.551054 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.551059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.551415 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.551533 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.551539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.552313 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.552316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.553006 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.553014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.553590 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.553592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.553663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.554587 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.554606 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.556170 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.557782 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.557786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.557790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.557976 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.558206 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.558209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.558211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.558289 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.558659 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.558665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.558806 (+ 141us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.558941 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.559234 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":96,"thread_start_us":132,"threads_started":1}
I20260618 22:31:27.560221 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.546906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.16.243.188:45359)
0618 22:31:27.547297 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.547313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.547349 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:31:27.547983 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.548002 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.548327 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.548614 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.549512 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.550865 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.551844 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.552335 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.552488 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.555030 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.555036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.555045 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.555065 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.556946 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.557409 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.557413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.557418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.557520 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.558054 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.558056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.558057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.558497 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.558714 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.558891 (+ 177us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.559175 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260618 22:31:27.561517 7374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:27.569959 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.561985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:45563 (local address 127.0.0.1:51444)
0618 22:31:27.562103 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.562116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.562182 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.562403 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.562406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.562603 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.562718 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.562726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.563627 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.563630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.564173 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.564183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.564618 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.564621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.564718 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.565209 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.565221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.566586 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.568612 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.568615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.568617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.568800 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.569054 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.569057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.569059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.569135 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.569472 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.569483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.569569 (+ 86us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.569650 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.569786 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 22:31:27.569959 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.562076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.16.243.189:45563)
0618 22:31:27.562196 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.562199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.562227 (+ 28us) server_negotiation.cc:407] Connection header received
0618 22:31:27.562264 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.562266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.562303 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.562357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.562891 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.563522 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.564289 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.564526 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.564600 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.567209 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.567212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.567214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.567239 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.568510 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.568886 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.568889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.568891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.568965 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.569212 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.569215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.569217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.569387 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.569469 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.569708 (+ 239us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.569819 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 22:31:27.570298 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.562732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.16.243.190:37163)
0618 22:31:27.562833 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.562835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.562849 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:31:27.562969 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.562971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.563013 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.563082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.563647 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.564104 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.564942 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.565253 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.565328 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.567549 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.567552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.567555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.567582 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.568861 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.569245 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.569249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.569250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.569321 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.569563 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.569566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.569568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.569734 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.569946 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.570057 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.570159 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:31:27.571076 7374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
I20260618 22:31:27.579560 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.571519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48156)
0618 22:31:27.571639 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.571652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.571713 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.571985 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.571988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.572164 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.572263 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.572271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.573200 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.573202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.573985 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.573995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.574438 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.574441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.574498 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.574956 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.574969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.576277 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.577984 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.577988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.577990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.578228 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.578493 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.578495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.578496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.578612 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.579004 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.579007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.579110 (+ 103us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.579218 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.579362 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 22:31:27.579604 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.571549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.16.243.188:45359)
0618 22:31:27.571809 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.571813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.571821 (+ 8us) server_negotiation.cc:407] Connection header received
0618 22:31:27.571846 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.571848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.571883 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.571951 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.572366 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.573085 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.574087 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.574329 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.574419 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.576418 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.576422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.576425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.576450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.577869 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.578314 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.578317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.578319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.578409 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.578703 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.578707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.578708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.578922 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.579018 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.579265 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.579436 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":210}
I20260618 22:31:27.579905 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.571524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.16.243.189:45563)
0618 22:31:27.571641 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.571644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.571788 (+ 144us) server_negotiation.cc:407] Connection header received
0618 22:31:27.571852 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.571855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.571891 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.571969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.572581 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.573267 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.574284 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.574594 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.574670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.576839 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.576842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.576845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.576869 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.578170 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.578652 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.578656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.578657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.578738 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.578978 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.578981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.578982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.579171 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.579368 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.579504 (+ 136us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.579624 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:31:27.583143 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: Bootstrap starting.
I20260618 22:31:27.583195 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:27.584381 7197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "558524616c5b4f788c1c28006d2a141c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56074
I20260618 22:31:27.585763 7374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:27.586635 7374 log.cc:824] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:27.588750 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: No bootstrap required, opened a new log
I20260618 22:31:27.590291 7223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 3
I20260618 22:31:27.592650 7374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.592867 7374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699a27a30f63419998ba0caf411355f9, State: Initialized, Role: FOLLOWER
I20260618 22:31:27.593222 7374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.593562 7298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } attempt: 3
I20260618 22:31:27.593746 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.593885 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.594695 7374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: configured and running, proceeding with master startup.
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:31:27 dist-test-slave-jm5g krb5kdc[7142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
W20260618 22:31:27.600611 7392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:27.600685 7392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:27.598423 7305 master_runner.cc:186] Error getting master registration for 127.16.243.190:37163: OK, instance_id { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" instance_seqno: 1781821887128963 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:31:27.601634 7305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:31:27.603775 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.590724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48164)
0618 22:31:27.590904 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.590920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.591014 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.591257 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.591260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.591457 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.591559 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.591566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.592530 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.592534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.593245 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.593254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.594408 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.594412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.594504 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.595040 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.595055 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.596527 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.602204 (+ 5677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.602208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.602210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.602408 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.602719 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.602722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.602724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.602828 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.603279 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.603281 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.603368 (+ 87us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.603457 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.603613 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260618 22:31:27.603899 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.590771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.16.243.188:45359)
0618 22:31:27.590919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.590924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.590966 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:31:27.591105 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.591109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.591155 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.591243 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.591668 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.592424 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.593356 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.594302 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.594400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.596720 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.596724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.596728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.596752 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.602111 (+ 5359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.602507 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.602510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.602512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.602617 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.602929 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.602932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.602933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.603158 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.603234 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.603675 (+ 441us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.603777 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:31:27.605322 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.593991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:45359 (local address 127.0.0.1:48172)
0618 22:31:27.594134 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:27.594149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:27.594254 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:27.595108 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:27.595112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:27.595314 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:27.595416 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.595424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.597534 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.597538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.598341 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:27.598359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.598936 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.598940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:27.599024 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.599466 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:27.599481 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:27.600884 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:27.603233 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.603236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.603238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.603481 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.604391 (+ 910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:27.604394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:27.604396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:27.604487 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:27.604825 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:27.604828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:27.604933 (+ 105us) client_negotiation.cc:769] Sending connection context
0618 22:31:27.605041 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:31:27.605178 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:31:27.605542 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.594227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.16.243.188:45359)
0618 22:31:27.594867 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.594876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.594915 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:31:27.594948 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.594951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.594992 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.595076 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.596291 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.597358 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.598481 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.598824 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.598916 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.601070 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.601073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.601076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.601099 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.603122 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.604181 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.604183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.604185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.604288 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.604572 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.604574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.604576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.604721 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.604783 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.605106 (+ 323us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.605218 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":104,"threads_started":1}
I20260618 22:31:27.607079 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: Bootstrap starting.
I20260618 22:31:27.608325 7298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: Bootstrap starting.
I20260618 22:31:27.609895 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:27.610757 7298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:27.610807 7223 log.cc:824] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:27.611526 7298 log.cc:824] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:27.613111 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: No bootstrap required, opened a new log
I20260618 22:31:27.613497 7298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: No bootstrap required, opened a new log
I20260618 22:31:27.616168 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.616398 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558524616c5b4f788c1c28006d2a141c, State: Initialized, Role: FOLLOWER
I20260618 22:31:27.616724 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.617249 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.617221 7298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.617422 7298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4cb8acb3bb4ee59bc70fd478d9fbd0, State: Initialized, Role: FOLLOWER
I20260618 22:31:27.617549 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.617722 7298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.617997 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:27.618086 7197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:31:27.618188 7197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:27.618955 7298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:27.619395 7197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.620342 7197 leader_election.cc:288] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [CANDIDATE]: Term 1 election: Requested vote from peers 9c4cb8acb3bb4ee59bc70fd478d9fbd0 (127.16.243.189:45563), 699a27a30f63419998ba0caf411355f9 (127.16.243.188:45359)
I20260618 22:31:27.620829 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.620913 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558524616c5b4f788c1c28006d2a141c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "699a27a30f63419998ba0caf411355f9"
I20260618 22:31:27.621034 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.621186 7348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:27.621177 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558524616c5b4f788c1c28006d2a141c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0"
I20260618 22:31:27.621359 7272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:27.622489 7348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558524616c5b4f788c1c28006d2a141c in term 1.
I20260618 22:31:27.622489 7272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558524616c5b4f788c1c28006d2a141c in term 1.
I20260618 22:31:27.622948 7172 leader_election.cc:302] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 558524616c5b4f788c1c28006d2a141c, 9c4cb8acb3bb4ee59bc70fd478d9fbd0; no voters:
I20260618 22:31:27.623416 7396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:27.623629 7396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [term 1 LEADER]: Becoming Leader. State: Replica: 558524616c5b4f788c1c28006d2a141c, State: Running, Role: LEADER
I20260618 22:31:27.623948 7396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } }
I20260618 22:31:27.625691 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558524616c5b4f788c1c28006d2a141c. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.625793 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: This master's current role is: LEADER
W20260618 22:31:27.626266 7176 tablet.cc:2369] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:31:27.626696 7411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:27.626772 7411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:27.627413 7427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:27.628193 7427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:27.634718 7348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [term 1 FOLLOWER]: Refusing update from remote peer 558524616c5b4f788c1c28006d2a141c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:27.634761 7272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [term 1 FOLLOWER]: Refusing update from remote peer 558524616c5b4f788c1c28006d2a141c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:27.635391 7396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [LEADER]: Connected to new peer: Peer: permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:27.635788 7396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:31:27.636264 7419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:27.636336 7419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:27.641702 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558524616c5b4f788c1c28006d2a141c. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.641824 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.642601 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558524616c5b4f788c1c28006d2a141c. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.642764 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.645334 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.634135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.16.243.188:45359)
0618 22:31:27.634248 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.634253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.635995 (+ 1742us) server_negotiation.cc:407] Connection header received
0618 22:31:27.636184 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.636187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.636233 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.636283 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.637322 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.638123 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.639326 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.639685 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.639762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.641607 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.641612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.641614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.641643 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.643446 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.643951 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.643954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.643956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.644055 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.644369 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.644372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.644374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.644582 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.644690 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.645064 (+ 374us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.645181 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:31:27.656486 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.656726 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:27.656865 7428 mvcc.cc:205] Tried to move back new op lower bound from 7298342451744837632 to 7298342451711049728. Current Snapshot: MvccSnapshot[applied={T|T < 7298342451744837632}]
I20260618 22:31:27.657099 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.657217 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.657923 7434 mvcc.cc:205] Tried to move back new op lower bound from 7298342451744837632 to 7298342451711049728. Current Snapshot: MvccSnapshot[applied={T|T < 7298342451744837632}]
I20260618 22:31:27.658967 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.659090 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:27.659391 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.635391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.16.243.189:45563)
0618 22:31:27.635490 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.635494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.650809 (+ 15315us) server_negotiation.cc:407] Connection header received
0618 22:31:27.650971 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.650974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.651025 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.651073 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.652034 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.652797 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.653923 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.654263 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.654332 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.655920 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.655923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.655926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.655951 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.657499 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.658049 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.658053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.658054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.658136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.658436 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.658440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.658442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.658668 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.658765 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.659125 (+ 360us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.659241 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 22:31:27.660188 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "558524616c5b4f788c1c28006d2a141c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558524616c5b4f788c1c28006d2a141c" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 37163 } } peers { permanent_uuid: "9c4cb8acb3bb4ee59bc70fd478d9fbd0" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 45563 } } peers { permanent_uuid: "699a27a30f63419998ba0caf411355f9" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 45359 } } }
I20260618 22:31:27.660301 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:27.660459 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:27.634925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.16.243.190:37163)
0618 22:31:27.635157 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.635162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.646955 (+ 11793us) server_negotiation.cc:407] Connection header received
0618 22:31:27.647135 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.647138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.647187 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.647234 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.648061 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.648830 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.649917 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:27.650289 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:27.650369 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:27.652017 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:27.652020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:27.652023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:27.652049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:27.653584 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.654061 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.654065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.654067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.654153 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:27.654453 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:27.654457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:27.654459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:27.654688 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:27.655020 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:27.660195 (+ 5175us) server_negotiation.cc:299] Negotiation successful
0618 22:31:27.660312 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260618 22:31:27.660745 7433 mvcc.cc:205] Tried to move back new op lower bound from 7298342451744837632 to 7298342451711049728. Current Snapshot: MvccSnapshot[applied={T|T < 7298342451744837632}]
I20260618 22:31:27.665827 7427 catalog_manager.cc:1347] Generated new cluster ID: 3d6160dba7ec48029675edd3dc1b93a2
I20260618 22:31:27.665894 7427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:27.692672 7427 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:27.694190 7427 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:27.708624 7427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 558524616c5b4f788c1c28006d2a141c: Generated new TSK 0
I20260618 22:31:27.709539 7427 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:31:27.745536 7381 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.743140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.16.243.188:45359)
0618 22:31:27.743259 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.743263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.744400 (+ 1137us) server_negotiation.cc:407] Connection header received
0618 22:31:27.744624 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.744628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.744681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.744735 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.745386 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56302: BlockingRecv error: recv got EOF from 127.0.0.1:56302 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 22:31:27.747089 7227 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.742626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.16.243.190:37163)
0618 22:31:27.742758 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.742763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.746182 (+ 3419us) server_negotiation.cc:407] Connection header received
0618 22:31:27.746340 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.746343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.746391 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.746437 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.746962 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56094: BlockingRecv error: recv got EOF from 127.0.0.1:56094 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260618 22:31:27.748507 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.743826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.16.243.189:45563)
0618 22:31:27.743934 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.743938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.747687 (+ 3749us) server_negotiation.cc:407] Connection header received
0618 22:31:27.747841 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.747845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.747892 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.747940 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.748379 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41932: BlockingRecv error: recv got EOF from 127.0.0.1:41932 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:31:27.753118 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.751553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.16.243.189:45563)
0618 22:31:27.751661 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.751664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.752161 (+ 497us) server_negotiation.cc:407] Connection header received
0618 22:31:27.752373 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.752377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.752452 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.752518 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.753014 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41946: BlockingRecv error: recv got EOF from 127.0.0.1:41946 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:31:27.758122 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.756674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.16.243.189:45563)
0618 22:31:27.756772 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.756776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.757251 (+ 475us) server_negotiation.cc:407] Connection header received
0618 22:31:27.757425 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.757428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.757490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.757541 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.758024 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41948: BlockingRecv error: recv got EOF from 127.0.0.1:41948 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260618 22:31:27.763973 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.762436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.16.243.189:45563)
0618 22:31:27.762579 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.762583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.763066 (+ 483us) server_negotiation.cc:407] Connection header received
0618 22:31:27.763265 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.763268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.763319 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.763367 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.763864 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41962: BlockingRecv error: recv got EOF from 127.0.0.1:41962 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:31:27.772092 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.770442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.16.243.189:45563)
0618 22:31:27.770633 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.770637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.771154 (+ 517us) server_negotiation.cc:407] Connection header received
0618 22:31:27.771343 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.771346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.771396 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.771444 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.771981 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41966: BlockingRecv error: recv got EOF from 127.0.0.1:41966 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260618 22:31:27.784817 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.782976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.16.243.189:45563)
0618 22:31:27.783127 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.783132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.783750 (+ 618us) server_negotiation.cc:407] Connection header received
0618 22:31:27.783979 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.783983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.784041 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.784100 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.784693 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41974: BlockingRecv error: recv got EOF from 127.0.0.1:41974 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260618 22:31:27.806011 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.804097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.16.243.189:45563)
0618 22:31:27.804243 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.804248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.804934 (+ 686us) server_negotiation.cc:407] Connection header received
0618 22:31:27.805141 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.805145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.805200 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.805253 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.805889 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41978: BlockingRecv error: recv got EOF from 127.0.0.1:41978 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260618 22:31:27.842027 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.840868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.16.243.189:45563)
0618 22:31:27.840989 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.840994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.841210 (+ 216us) server_negotiation.cc:407] Connection header received
0618 22:31:27.841349 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.841351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.841398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.841473 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.841917 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260618 22:31:27.909148 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:27.908014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.16.243.189:45563)
0618 22:31:27.908169 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:27.908173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:27.908303 (+ 130us) server_negotiation.cc:407] Connection header received
0618 22:31:27.908490 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:27.908493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:27.908544 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:27.908619 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:27.909028 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:31:28.040256 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:28.039296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.16.243.189:45563)
0618 22:31:28.039411 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.039415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.039540 (+ 125us) server_negotiation.cc:407] Connection header received
0618 22:31:28.039665 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.039667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.039714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.039791 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:28.040151 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42000: BlockingRecv error: recv got EOF from 127.0.0.1:42000 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260618 22:31:28.299540 7304 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:28.298415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.16.243.189:45563)
0618 22:31:28.298572 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.298577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.298704 (+ 127us) server_negotiation.cc:407] Connection header received
0618 22:31:28.298828 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.298831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.298886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.298968 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:28.299425 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42016: BlockingRecv error: recv got EOF from 127.0.0.1:42016 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:31:28.601985 7392 catalog_manager.cc:1259] Loaded cluster ID: 3d6160dba7ec48029675edd3dc1b93a2
I20260618 22:31:28.602077 7392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: loading cluster ID for follower catalog manager: success
I20260618 22:31:28.603806 7392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: acquiring CA information for follower catalog manager: success
I20260618 22:31:28.605037 7392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 699a27a30f63419998ba0caf411355f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:31:28.637594 7419 catalog_manager.cc:1259] Loaded cluster ID: 3d6160dba7ec48029675edd3dc1b93a2
I20260618 22:31:28.637689 7419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: loading cluster ID for follower catalog manager: success
I20260618 22:31:28.639416 7419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: acquiring CA information for follower catalog manager: success
I20260618 22:31:28.640606 7419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c4cb8acb3bb4ee59bc70fd478d9fbd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:31:28.817579 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:28.813634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.16.243.189:45563)
0618 22:31:28.813838 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.813841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.813856 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:31:28.814048 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.814052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.814184 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.814292 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:28.814567 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.815113 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.816057 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.816295 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.816384 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:28.816640 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:28.817093 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:28.817186 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:28.817334 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 22:31:28.817398 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260618 22:31:28.824395 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:28.820677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.16.243.188:45359)
0618 22:31:28.820907 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.820910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.820923 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:31:28.821100 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.821103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.821221 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.821321 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:28.821532 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.822050 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.822949 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.823189 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.823276 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:28.823565 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:28.823990 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:28.824098 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:28.824195 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 22:31:28.824250 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":57,"threads_started":1}
I20260618 22:31:28.831174 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:28.827152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.16.243.188:45359)
0618 22:31:28.827256 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.827260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.827496 (+ 236us) server_negotiation.cc:407] Connection header received
0618 22:31:28.827976 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.827979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.828030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.828095 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:28.828416 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.828925 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830065 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.830312 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830409 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:28.830701 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:28.830786 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:28.830845 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:28.830963 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 22:31:28.831030 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:31:28.831372 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:28.827464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.16.243.189:45563)
0618 22:31:28.827589 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.827594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.827792 (+ 198us) server_negotiation.cc:407] Connection header received
0618 22:31:28.827881 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.827883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.827937 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.828008 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:28.828263 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.829054 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830092 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.830411 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:28.830942 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:28.831019 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:28.831076 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:28.831205 (+ 129us) server_negotiation.cc:299] Negotiation successful
0618 22:31:28.831261 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:31:28.832020 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:28.827154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.16.243.190:37163)
0618 22:31:28.827385 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:28.827389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:28.827705 (+ 316us) server_negotiation.cc:407] Connection header received
0618 22:31:28.827874 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:28.827878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:28.828061 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:28.828175 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:28.828464 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.829239 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830307 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:28.830691 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:28.830794 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:28.831146 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:28.831466 (+ 320us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:28.831562 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:28.831817 (+ 255us) server_negotiation.cc:299] Negotiation successful
0618 22:31:28.831884 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260618 22:31:28.835134 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7155
I20260618 22:31:28.840166 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7230
I20260618 22:31:28.845062 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7305
2026-06-18T22:31:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2960 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:31:28 dist-test-slave-jm5g krb5kdc[7522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:31:28 dist-test-slave-jm5g krb5kdc[7522](info): set up 2 sockets
Jun 18 22:31:28 dist-test-slave-jm5g krb5kdc[7522](info): commencing operation
krb5kdc: starting...
W20260618 22:31:29.796052 17358 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/test-user.keytab.
Jun 18 22:31:29 dist-test-slave-jm5g krb5kdc[7522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:31:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
Entry for principal kudu/127.16.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab.
I20260618 22:31:29.911718 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41381
--webserver_interface=127.16.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:31:30.022078 7535 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:30.022405 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:30.022451 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:30.022514 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:30.026597 7535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:30.026674 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:30.026700 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:30.026722 7535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:30.026751 7535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:30.032200 7535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.190.keytab
--principal=kudu/127.16.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.190:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:30.033329 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:30.034335 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:30.043340 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:30.043332 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:30.043334 7535 server_base.cc:1034] running on GCE node
W20260618 22:31:30.043367 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:30.044140 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:30.045253 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:30.046424 7535 hybrid_clock.cc:648] HybridClock initialized: now 1781821890046405 us; error 38 us; skew 500 ppm
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:30.049182 7535 init.cc:377] Logged in from keytab as kudu/127.16.243.190@KRBTEST.COM (short username kudu)
I20260618 22:31:30.050122 7535 webserver.cc:460] Webserver started at http://127.16.243.190:36403/ using document root <none> and password file <none>
I20260618 22:31:30.050706 7535 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:30.050781 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:30.050984 7535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:30.052986 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data/instance:
uuid: "a51961ef5d844d44995d92edade83d74"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "94354071fd69d5f125bc6756e16ebab7"
server_key_iv: "c79db32a5c70ee62752a1496d17e53a7"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.053514 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance:
uuid: "a51961ef5d844d44995d92edade83d74"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "94354071fd69d5f125bc6756e16ebab7"
server_key_iv: "c79db32a5c70ee62752a1496d17e53a7"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.057183 7535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:31:30.059643 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:30.060890 7535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:30.061019 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal
uuid: "a51961ef5d844d44995d92edade83d74"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "94354071fd69d5f125bc6756e16ebab7"
server_key_iv: "c79db32a5c70ee62752a1496d17e53a7"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.061124 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:30.085142 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:30.088469 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:30.088655 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:30.095296 7535 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.190:41381
I20260618 22:31:30.095552 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.190:41381 every 8 connection(s)
I20260618 22:31:30.096256 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/data/info.pb
I20260618 22:31:30.097849 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7535
I20260618 22:31:30.097971 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-0/wal/instance
I20260618 22:31:30.098263 17358 external_mini_cluster.cc:1428] Setting key be1f6a5bd743ffdb0f964d7ccb44909d
I20260618 22:31:30.099372 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:30.103343 7603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.189@KRBTEST.COM" created.
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:30.118610 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.105796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41381 (local address 127.0.0.1:58342)
0618 22:31:30.106343 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.106389 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.107321 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.108038 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.108043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.108465 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.108747 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.108761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.109861 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.109865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.110674 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.110683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.111290 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.111294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.111431 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.112026 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.112042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.113704 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.115651 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.115659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.115670 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.115881 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.116189 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.116192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.116194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.116299 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.116870 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.116877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.117015 (+ 138us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.117167 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.117549 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":99,"thread_start_us":130,"threads_started":1}
I20260618 22:31:30.118610 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.106619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.16.243.190:41381)
0618 22:31:30.106853 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.106862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.106897 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:31:30.107616 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.107621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.107817 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.108016 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.108881 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.109745 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.110784 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.111145 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.111283 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.113853 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.113859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.113870 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.113891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.115515 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.115996 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.116000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.116002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.116106 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.116378 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.116381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.116383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.116765 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.116884 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.117249 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.117481 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
Entry for principal kudu/127.16.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab.
I20260618 22:31:30.121042 7603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:30.121639 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:42307
--webserver_interface=127.16.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:31:30.121804 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.121488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54796)
0618 22:31:30.121628 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.121720 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260618 22:31:30.122052 7553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
I20260618 22:31:30.122931 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.122721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54812)
0618 22:31:30.122826 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.122875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:31:30.123059 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:42307: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
I20260618 22:31:30.172338 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } attempt: 1
I20260618 22:31:30.173204 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.172917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54822)
0618 22:31:30.173061 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.173124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:31:30.174021 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.173807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54834)
0618 22:31:30.173898 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.173955 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:31:30.174144 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:42307: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
W20260618 22:31:30.231621 7610 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:30.231920 7610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:30.231943 7610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:30.231990 7610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:31:30.236028 7610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:30.236095 7610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:30.236114 7610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:30.236129 7610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:30.236145 7610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:30.241598 7610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.189.keytab
--principal=kudu/127.16.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.189:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:30.242307 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } attempt: 2
I20260618 22:31:30.242770 7610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:30.243153 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.242862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54848)
0618 22:31:30.243005 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.243068 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:31:30.243800 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.243598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54852)
0618 22:31:30.243698 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.243742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:31:30.243906 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.189:42307: Network error: Client connection negotiation failed: client connection to 127.16.243.189:42307: connect: Connection refused (error 111)
I20260618 22:31:30.243906 7610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:31:30.251919 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:30.251866 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:30.251896 7618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:30.252071 7610 server_base.cc:1034] running on GCE node
I20260618 22:31:30.252660 7610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:30.253732 7610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:30.254918 7610 hybrid_clock.cc:648] HybridClock initialized: now 1781821890254898 us; error 33 us; skew 500 ppm
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:30.257676 7610 init.cc:377] Logged in from keytab as kudu/127.16.243.189@KRBTEST.COM (short username kudu)
I20260618 22:31:30.258621 7610 webserver.cc:460] Webserver started at http://127.16.243.189:39667/ using document root <none> and password file <none>
I20260618 22:31:30.259162 7610 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:30.259227 7610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:30.259423 7610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:30.261514 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data/instance:
uuid: "74cd8eef402f45cbafa71bef9303de50"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "1ac896c1a29850b523175065da005e63"
server_key_iv: "2579d31021da2897e7a3bdac19feaf77"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.262033 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal/instance:
uuid: "74cd8eef402f45cbafa71bef9303de50"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "1ac896c1a29850b523175065da005e63"
server_key_iv: "2579d31021da2897e7a3bdac19feaf77"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.265895 7610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:31:30.268321 7625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:30.269582 7610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:31:30.269690 7610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal
uuid: "74cd8eef402f45cbafa71bef9303de50"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "1ac896c1a29850b523175065da005e63"
server_key_iv: "2579d31021da2897e7a3bdac19feaf77"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.269800 7610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:30.301175 7610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:30.304090 7610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:30.304263 7610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:30.311170 7610 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.189:42307
I20260618 22:31:30.311429 7677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.189:42307 every 8 connection(s)
I20260618 22:31:30.312166 7610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/data/info.pb
I20260618 22:31:30.314985 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:30.318210 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7610
I20260618 22:31:30.318331 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-1/wal/instance
I20260618 22:31:30.318657 17358 external_mini_cluster.cc:1428] Setting key 30e2bceb88b27a9f093d7a4ff02a7449
I20260618 22:31:30.319135 7678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.243.188@KRBTEST.COM" created.
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:30.333616 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.321337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.16.243.190:41381)
0618 22:31:30.321468 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.321472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.322673 (+ 1201us) server_negotiation.cc:407] Connection header received
0618 22:31:30.323686 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.323689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.323754 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.323826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.325001 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.325527 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.326722 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.326998 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.327095 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.329682 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.329687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.329690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.329725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.331264 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.331881 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.331884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.331885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.331977 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.332347 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.332364 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.332366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.332603 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.332689 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.333366 (+ 677us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.333486 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:31:30.334869 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.321893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41381 (local address 127.0.0.1:58346)
0618 22:31:30.322480 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.322528 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.323465 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.324005 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.324012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.324370 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.324848 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.324861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.325647 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.325650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.326603 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.326612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.327148 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.327152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.327306 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.327832 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.327848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.329465 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.331394 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.331401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.331420 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.331775 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.332066 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.332069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.332071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.332259 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.332711 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.332719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.333081 (+ 362us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.333314 (+ 233us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.333709 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":156,"threads_started":1}
I20260618 22:31:30.335038 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } attempt: 3
I20260618 22:31:30.337004 7678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
Entry for principal kudu/127.16.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab.
I20260618 22:31:30.343914 17358 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9_akNk/build/debug/bin/kudu
/tmp/dist-test-task9_akNk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:36673
--webserver_interface=127.16.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.243.148:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:31:30.345806 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.337557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.16.243.189:42307)
0618 22:31:30.337875 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.337881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.337892 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:31:30.337919 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.337921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.337951 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.338034 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.338385 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.339063 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.339806 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.340116 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.340206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.341894 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.341904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.341922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.341945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.344059 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344443 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.344449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.344451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.344544 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344769 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.344772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.344774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.345087 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.345205 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.345457 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.345628 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":156,"threads_started":1}
I20260618 22:31:30.345885 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.335506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54868)
0618 22:31:30.335628 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.335642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.335719 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.336447 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.336450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.336625 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.336722 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.336727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.337899 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.337903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.338971 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.338981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.339599 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.339603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.339704 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.340163 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.340174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.341775 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.344226 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.344231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.344541 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.344855 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.344860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.344948 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.345401 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.345404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.345502 (+ 98us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.345607 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.345744 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:31:30.346019 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.335598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.16.243.189:42307)
0618 22:31:30.336086 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.336099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.336118 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:31:30.336151 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.336155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.336327 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.336454 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.336823 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.337796 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.339076 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.339495 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.339586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.341961 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.341964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.341966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.341989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.344110 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344686 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.344689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.344691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.344777 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.345054 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.345057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.345059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.345309 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.345407 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.345672 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.345791 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"spinlock_wait_cycles":896,"thread_start_us":128,"threads_started":1}
I20260618 22:31:30.346426 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.337471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54872)
0618 22:31:30.337564 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.337576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.337671 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.338058 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.338061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.338211 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.338288 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.338292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.339156 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.339159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.339708 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.339717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.340205 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.340208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.340286 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.340651 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.340665 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.341766 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.344169 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.344174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.344348 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.344621 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.344624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.344626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.344699 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.345177 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.345179 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.345263 (+ 84us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.346150 (+ 887us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.346279 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"spinlock_wait_cycles":5888}
I20260618 22:31:30.348039 7678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:30.348215 7603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:30.348830 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.348601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53702)
0618 22:31:30.348716 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.348768 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:31:30.349141 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.348618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53696)
0618 22:31:30.349002 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.349079 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
W20260618 22:31:30.349313 7626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
I20260618 22:31:30.349570 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.349295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53712)
0618 22:31:30.349451 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.349510 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260618 22:31:30.349706 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
I20260618 22:31:30.350303 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.350089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53726)
0618 22:31:30.350189 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.350242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:31:30.350432 7678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
I20260618 22:31:30.380894 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 1
I20260618 22:31:30.381713 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.381423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53728)
0618 22:31:30.381572 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.381633 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 22:31:30.382483 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.382273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53732)
0618 22:31:30.382379 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.382425 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:31:30.382632 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
I20260618 22:31:30.399739 7678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 1
I20260618 22:31:30.400629 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.400298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53742)
0618 22:31:30.400455 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.400540 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 22:31:30.401367 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.401151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53750)
0618 22:31:30.401248 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.401299 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:31:30.401547 7678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
W20260618 22:31:30.456506 7685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:31:30.456816 7685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:31:30.456842 7685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:31:30.456903 7685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260618 22:31:30.457789 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 2
I20260618 22:31:30.458658 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.458316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53758)
0618 22:31:30.458473 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.458544 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 22:31:30.459335 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.459131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53774)
0618 22:31:30.459233 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.459279 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:31:30.459441 7603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
W20260618 22:31:30.461364 7685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:31:30.461421 7685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:31:30.461447 7685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:31:30.461478 7685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:31:30.461508 7685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:31:30.467128 7685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.243.148:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.243.190:41381,127.16.243.189:42307,127.16.243.188:36673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/krb5kdc/kudu_127.16.243.188.keytab
--principal=kudu/127.16.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.243.188:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:31:30.468200 7685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:31:30.469213 7685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:31:30.469707 7678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 2
I20260618 22:31:30.470466 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.470206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53782)
0618 22:31:30.470337 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.470395 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 22:31:30.471135 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.470927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53792)
0618 22:31:30.471038 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.471081 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:31:30.471257 7678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.243.188:36673: Network error: Client connection negotiation failed: client connection to 127.16.243.188:36673: connect: Connection refused (error 111)
W20260618 22:31:30.477936 7694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:30.477983 7685 server_base.cc:1034] running on GCE node
W20260618 22:31:30.478022 7692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:31:30.477936 7691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:31:30.478761 7685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:31:30.479888 7685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:31:30.481046 7685 hybrid_clock.cc:648] HybridClock initialized: now 1781821890481023 us; error 38 us; skew 500 ppm
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:31:30.483870 7685 init.cc:377] Logged in from keytab as kudu/127.16.243.188@KRBTEST.COM (short username kudu)
I20260618 22:31:30.484786 7685 webserver.cc:460] Webserver started at http://127.16.243.188:44045/ using document root <none> and password file <none>
I20260618 22:31:30.485324 7685 fs_manager.cc:359] Metadata directory not provided
I20260618 22:31:30.485386 7685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:31:30.485587 7685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:31:30.487679 7685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data/instance:
uuid: "389219b5bfe94af3b56799c2eab1e67f"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "29aaff5f10159b08412f2277c8cfae11"
server_key_iv: "f2c0dc2319d4a8e96c881f147affd10a"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.488206 7685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal/instance:
uuid: "389219b5bfe94af3b56799c2eab1e67f"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "29aaff5f10159b08412f2277c8cfae11"
server_key_iv: "f2c0dc2319d4a8e96c881f147affd10a"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.491853 7685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:31:30.494199 7701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:31:30.495430 7685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:31:30.495545 7685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data,/tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal
uuid: "389219b5bfe94af3b56799c2eab1e67f"
format_stamp: "Formatted at 2026-06-18 22:31:30 on dist-test-slave-jm5g"
server_key: "29aaff5f10159b08412f2277c8cfae11"
server_key_iv: "f2c0dc2319d4a8e96c881f147affd10a"
server_key_version: "encryptionkey@0"
I20260618 22:31:30.495661 7685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:31:30.523411 7685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:31:30.526963 7685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:31:30.527163 7685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:31:30.535200 7685 rpc_server.cc:307] RPC server started. Bound to: 127.16.243.188:36673
I20260618 22:31:30.535476 7753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.243.188:36673 every 8 connection(s)
I20260618 22:31:30.536227 7685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/data/info.pb
I20260618 22:31:30.539109 7754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:31:30.541150 17358 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9_akNk/build/debug/bin/kudu as pid 7685
I20260618 22:31:30.541276 17358 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9_akNk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821616719733-17358-0/minicluster-data/master-2/wal/instance
I20260618 22:31:30.541584 17358 external_mini_cluster.cc:1428] Setting key 0380d5753a3fb1226b05085de2e5843b
I20260618 22:31:30.543682 7754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:30.555274 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.546146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.16.243.190:41381)
0618 22:31:30.546272 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.546277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.546574 (+ 297us) server_negotiation.cc:407] Connection header received
0618 22:31:30.546747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.546749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.546790 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.546853 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.547626 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.548104 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.549215 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.549487 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.549597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.551937 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.551940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.551942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.551966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.553294 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.553789 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.553792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.553794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.553897 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.554207 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.554211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.554212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.554459 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.554587 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.554971 (+ 384us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.555109 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:31:30.556200 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.542968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.16.243.188:36673)
0618 22:31:30.543404 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.543429 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.543473 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:31:30.544212 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.544237 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.544706 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.545088 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.545827 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.547387 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.548238 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.548928 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.549089 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.551512 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.551518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.551549 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.551570 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.553304 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.553788 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.553793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.553795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.553938 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.554207 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.554210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.554212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.554729 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.554887 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.555109 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.555335 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260618 22:31:30.556197 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.546035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.190:41381 (local address 127.0.0.1:58358)
0618 22:31:30.546414 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.546466 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.546654 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.546918 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.546924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.547365 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.547508 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.547515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.548217 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.548220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.549097 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.549105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.549639 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.549644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.549731 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.550164 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.550180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.551798 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.553407 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.553416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.553427 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.553706 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.553982 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.553984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.553986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.554128 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.554595 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.554602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.554739 (+ 137us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.554912 (+ 173us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.555250 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260618 22:31:30.557750 7754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:30.558791 7603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 3
I20260618 22:31:30.562399 7678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } attempt: 3
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.190@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.190@KRBTEST.COM
I20260618 22:31:30.568468 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.558515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.16.243.190:41381)
0618 22:31:30.558762 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.558766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.558778 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:31:30.558813 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.558816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.558856 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.558923 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.559715 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.560381 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561389 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.561663 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561744 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.564811 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.564815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.564817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.564840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.566339 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566856 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.566860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.566861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.566945 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.567228 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.567233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.567235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.567452 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.567685 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.567995 (+ 310us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.568266 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":174}
I20260618 22:31:30.568511 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.559194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53808)
0618 22:31:30.559281 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.559294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.559365 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.559643 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.559646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.559792 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.559863 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.559868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.560543 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.560546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.561034 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.561041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.561563 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.561663 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.562128 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.562143 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.564607 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.566966 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.566971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.567199 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.567486 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.567490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.567492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.567576 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.567988 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.567992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.568122 (+ 130us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.568202 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.568328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":5}
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.189@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
I20260618 22:31:30.568768 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.558320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.16.243.189:42307)
0618 22:31:30.558445 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.558450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.558468 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:31:30.558510 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.558513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.558616 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.558698 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.559330 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.560154 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561178 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.561526 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561619 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.564188 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.564191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.564194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.564219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.566096 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566619 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.566623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.566625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.566708 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566988 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.566991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.566993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.567394 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.567482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.568494 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.568629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":39}
I20260618 22:31:30.569463 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.558235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.189:42307 (local address 127.0.0.1:54882)
0618 22:31:30.558338 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.558352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.558430 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.558849 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.558852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.559061 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.559183 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.559191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.560281 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.560284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.561081 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.561090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.561718 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.561785 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.562265 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.562279 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.564034 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.566211 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.566217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.566483 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.566807 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.566810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.566812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.566900 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.568221 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.568224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.568312 (+ 88us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.569137 (+ 825us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.569311 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260618 22:31:30.570201 7754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:31:30.570889 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.559398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.16.243.188:36673)
0618 22:31:30.559486 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.559489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.559497 (+ 8us) server_negotiation.cc:407] Connection header received
0618 22:31:30.559520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.559522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.559551 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.559609 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.559960 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.560451 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561130 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.561455 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.561549 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.564773 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.564777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.564779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.564802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.566865 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.567309 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.567312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.567313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.567387 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.567689 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.567692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.567694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.567881 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.570562 (+ 2681us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.570642 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.570747 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":31488}
I20260618 22:31:30.573074 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.562933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53814)
0618 22:31:30.563224 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.563239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.563317 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.563787 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.563790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.563948 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.564023 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.564037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.565539 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.565543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.566345 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.566354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.566999 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.567002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.567132 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.567530 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.567544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.568965 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.571521 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.571524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.571527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.571774 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.572055 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.572059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.572061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.572150 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.572589 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.572591 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.572704 (+ 113us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.572779 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.572919 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":4}
I20260618 22:31:30.573763 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.563200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.16.243.188:36673)
0618 22:31:30.563589 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.563595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.563605 (+ 10us) server_negotiation.cc:407] Connection header received
0618 22:31:30.563636 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.563639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.563678 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.563756 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.564169 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.565412 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.566504 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.566877 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.567533 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.569096 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.569099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.569102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.569127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.571415 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.571876 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.571879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.571881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.571961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.572253 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.572256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.572258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.572440 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.573341 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.573414 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.573523 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":186,"threads_started":1}
I20260618 22:31:30.574515 7603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: Bootstrap starting.
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.243.189@KRBTEST.COM
Jun 18 22:31:30 dist-test-slave-jm5g krb5kdc[7522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.243.188@KRBTEST.COM for kudu/127.16.243.188@KRBTEST.COM
I20260618 22:31:30.576946 7678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: Bootstrap starting.
I20260618 22:31:30.577643 7603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:30.578624 7603 log.cc:824] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:30.579802 7678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:30.580202 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.570689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.243.188:36673 (local address 127.0.0.1:53824)
0618 22:31:30.570956 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 22:31:30.570969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:31:30.571029 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:31:30.571330 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:31:30.571333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:31:30.571489 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:31:30.571578 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.571585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.572470 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.572473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.573218 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:31:30.573228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.573834 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.573837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:31:30.573900 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.574340 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:31:30.574353 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:31:30.576641 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:31:30.578761 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.578763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.578765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.578990 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.579258 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:31:30.579261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:31:30.579263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:31:30.579332 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:31:30.579711 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:31:30.579714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:31:30.579797 (+ 83us) client_negotiation.cc:769] Sending connection context
0618 22:31:30.579890 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 22:31:30.580044 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260618 22:31:30.580550 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.569720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.16.243.189:42307)
0618 22:31:30.569819 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.569823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.569878 (+ 55us) server_negotiation.cc:407] Connection header received
0618 22:31:30.570114 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.570117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.570161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.570228 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.570957 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.571673 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.572681 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.572969 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.573047 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.576412 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.576416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.576419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.576444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.578146 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.578749 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.578752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.578754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.578838 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.579209 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.579213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.579215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.579437 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.579630 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.580054 (+ 424us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.580172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:31:30.580775 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.570737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.16.243.188:36673)
0618 22:31:30.571010 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.571017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.571030 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:31:30.571096 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.571098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.571139 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.571218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.571682 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.572368 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.573327 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.573658 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.573743 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.576767 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.576770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.576772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.576794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.578654 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.579102 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.579105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.579106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.579180 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.579405 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.579407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.579409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.579595 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.579680 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.580513 (+ 833us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.580621 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
I20260618 22:31:30.580735 7678 log.cc:824] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: Log is configured to *not* fsync() on all Append() calls
I20260618 22:31:30.581037 7603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: No bootstrap required, opened a new log
I20260618 22:31:30.582943 7678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: No bootstrap required, opened a new log
I20260618 22:31:30.584983 17358 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:31:30.584829 7603 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.585080 7603 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51961ef5d844d44995d92edade83d74, State: Initialized, Role: FOLLOWER
I20260618 22:31:30.585402 7603 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.586199 7577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a51961ef5d844d44995d92edade83d74"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58372
I20260618 22:31:30.586215 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.586388 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.586443 7678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.586683 7678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cd8eef402f45cbafa71bef9303de50, State: Initialized, Role: FOLLOWER
I20260618 22:31:30.586935 7603 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:30.587009 7577 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:31:30.587004 7678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.587103 7577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:30.588346 7678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:30.588384 7577 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.589161 7754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: Bootstrap starting.
I20260618 22:31:30.589349 7577 leader_election.cc:288] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [CANDIDATE]: Term 1 election: Requested vote from peers 74cd8eef402f45cbafa71bef9303de50 (127.16.243.189:42307), 389219b5bfe94af3b56799c2eab1e67f (127.16.243.188:36673)
I20260618 22:31:30.589874 7650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a51961ef5d844d44995d92edade83d74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74cd8eef402f45cbafa71bef9303de50"
I20260618 22:31:30.590114 7650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:30.591490 7650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51961ef5d844d44995d92edade83d74 in term 1.
W20260618 22:31:30.591811 7556 tablet.cc:2369] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:31:30.591995 7754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: Neither blocks nor log segments found. Creating new log.
I20260618 22:31:30.592561 7553 leader_election.cc:302] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74cd8eef402f45cbafa71bef9303de50, a51961ef5d844d44995d92edade83d74; no voters:
I20260618 22:31:30.592882 7754 log.cc:824] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: Log is configured to *not* fsync() on all Append() calls
W20260618 22:31:30.595110 7775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:30.595187 7775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:30.595196 7754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: No bootstrap required, opened a new log
I20260618 22:31:30.592973 7535 master_runner.cc:186] Error getting master registration for 127.16.243.189:42307: OK, instance_id { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" instance_seqno: 1781821890309258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:31:30.596917 7535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:31:30.598642 7763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:31:30.598855 7763 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [term 1 LEADER]: Becoming Leader. State: Replica: a51961ef5d844d44995d92edade83d74, State: Running, Role: LEADER
I20260618 22:31:30.599255 7763 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.600291 7754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.600572 7754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389219b5bfe94af3b56799c2eab1e67f, State: Initialized, Role: FOLLOWER
I20260618 22:31:30.600893 7754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } }
I20260618 22:31:30.601380 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51961ef5d844d44995d92edade83d74. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.601508 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:30.607415 7798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:31:30.608479 7798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:31:30.610509 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.595643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.16.243.188:36673)
0618 22:31:30.595765 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.595770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.600143 (+ 4373us) server_negotiation.cc:407] Connection header received
0618 22:31:30.601453 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.601456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.601509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.601557 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.602467 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.603264 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.604303 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.604643 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.604721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.606253 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.606257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.606260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.606290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.607965 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.608768 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.608783 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.608786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.608879 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.609295 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.609299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.609301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.609519 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.609659 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.610186 (+ 527us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.610334 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:31:30.607385 7610 master_runner.cc:186] Error getting master registration for 127.16.243.188:36673: OK, instance_id { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" instance_seqno: 1781821890533008 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:31:30.610842 7610 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:31:30.613216 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.596027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.16.243.189:42307)
0618 22:31:30.596141 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.596145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.596794 (+ 649us) server_negotiation.cc:407] Connection header received
0618 22:31:30.597249 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.597252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.597299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.597345 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.598490 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.599255 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.601298 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.601613 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.601683 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.604202 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.604207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.604209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.604239 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.606145 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.611881 (+ 5736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.611885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.611887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.611986 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.612291 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.612294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.612295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.612503 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.612615 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.612957 (+ 342us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.613073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:31:30.613873 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:30.595814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.16.243.190:41381)
0618 22:31:30.595916 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.595920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.597018 (+ 1098us) server_negotiation.cc:407] Connection header received
0618 22:31:30.597208 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.597212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.597260 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.597310 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.598238 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.599035 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.600454 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:30.600769 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:30.600842 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:30.610385 (+ 9543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:31:30.610389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:31:30.610393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:31:30.610422 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:31:30.612073 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.612566 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.612570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.612572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.612663 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:31:30.612963 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:31:30.612966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:31:30.612969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:31:30.613181 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:31:30.613274 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:30.613620 (+ 346us) server_negotiation.cc:299] Negotiation successful
0618 22:31:30.613735 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260618 22:31:30.614434 7799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:30.614498 7799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:30.616165 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.616283 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.616281 7754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:31:30.616534 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.616672 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.616780 7727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:31:30.618268 7727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [term 1 FOLLOWER]: Refusing update from remote peer a51961ef5d844d44995d92edade83d74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:30.619628 7728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a51961ef5d844d44995d92edade83d74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "389219b5bfe94af3b56799c2eab1e67f"
I20260618 22:31:30.620927 7728 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51961ef5d844d44995d92edade83d74 in term 1.
W20260618 22:31:30.621819 7812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:31:30.621888 7812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:31:30.622778 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:30.628010 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51961ef5d844d44995d92edade83d74. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.628165 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.629067 7650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [term 1 FOLLOWER]: Refusing update from remote peer a51961ef5d844d44995d92edade83d74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:31:30.629666 7795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:31:30.633998 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51961ef5d844d44995d92edade83d74. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.634163 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.638217 7800 mvcc.cc:205] Tried to move back new op lower bound from 7298342463957803008 to 7298342463899312128. Current Snapshot: MvccSnapshot[applied={T|T < 7298342463957803008}]
I20260618 22:31:30.639607 7815 mvcc.cc:205] Tried to move back new op lower bound from 7298342463957803008 to 7298342463899312128. Current Snapshot: MvccSnapshot[applied={T|T < 7298342463957803008}]
I20260618 22:31:30.639768 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.639874 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.640585 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.640710 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:30.640815 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.640885 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74 [sys.catalog]: This master's current role is: LEADER
I20260618 22:31:30.641439 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a51961ef5d844d44995d92edade83d74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51961ef5d844d44995d92edade83d74" member_type: VOTER last_known_addr { host: "127.16.243.190" port: 41381 } } peers { permanent_uuid: "74cd8eef402f45cbafa71bef9303de50" member_type: VOTER last_known_addr { host: "127.16.243.189" port: 42307 } } peers { permanent_uuid: "389219b5bfe94af3b56799c2eab1e67f" member_type: VOTER last_known_addr { host: "127.16.243.188" port: 36673 } } }
I20260618 22:31:30.641568 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:31:30.648214 7798 catalog_manager.cc:1347] Generated new cluster ID: 7c27456a8664437bb75ea3f6141254e6
I20260618 22:31:30.648293 7798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:31:30.698819 7798 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:31:30.700296 7798 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:31:30.713110 7798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51961ef5d844d44995d92edade83d74: Generated new TSK 0
I20260618 22:31:30.714083 7798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:31:30.747890 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.744540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.16.243.189:42307)
0618 22:31:30.744679 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.744684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.746745 (+ 2061us) server_negotiation.cc:407] Connection header received
0618 22:31:30.746948 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.746952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.747007 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.747061 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.747744 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54898: BlockingRecv error: recv got EOF from 127.0.0.1:54898 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260618 22:31:30.749323 7607 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.745375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.16.243.190:41381)
0618 22:31:30.745507 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.745511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.748389 (+ 2878us) server_negotiation.cc:407] Connection header received
0618 22:31:30.748575 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.748579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.748640 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.748690 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.749195 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58382: BlockingRecv error: recv got EOF from 127.0.0.1:58382 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 22:31:30.750886 7760 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.746107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.16.243.188:36673)
0618 22:31:30.746215 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.746219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.749845 (+ 3626us) server_negotiation.cc:407] Connection header received
0618 22:31:30.750086 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.750090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.750141 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.750189 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.750764 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53840: BlockingRecv error: recv got EOF from 127.0.0.1:53840 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:31:30.755511 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.753838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.16.243.189:42307)
0618 22:31:30.753948 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.753952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.754440 (+ 488us) server_negotiation.cc:407] Connection header received
0618 22:31:30.754733 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.754736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.754817 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.754873 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.755394 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54914: BlockingRecv error: recv got EOF from 127.0.0.1:54914 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:31:30.760882 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.759079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.16.243.189:42307)
0618 22:31:30.759196 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.759200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.759859 (+ 659us) server_negotiation.cc:407] Connection header received
0618 22:31:30.760063 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.760066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.760122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.760176 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.760769 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54920: BlockingRecv error: recv got EOF from 127.0.0.1:54920 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:31:30.766978 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.765144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.16.243.189:42307)
0618 22:31:30.765264 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.765268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.765909 (+ 641us) server_negotiation.cc:407] Connection header received
0618 22:31:30.766133 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.766137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.766194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.766251 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.766861 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54928: BlockingRecv error: recv got EOF from 127.0.0.1:54928 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:31:30.775524 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.773428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.16.243.189:42307)
0618 22:31:30.773618 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.773623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.774306 (+ 683us) server_negotiation.cc:407] Connection header received
0618 22:31:30.774656 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.774661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.774722 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.774782 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.775383 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54934: BlockingRecv error: recv got EOF from 127.0.0.1:54934 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260618 22:31:30.787881 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.786005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.16.243.189:42307)
0618 22:31:30.786139 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.786143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.786826 (+ 683us) server_negotiation.cc:407] Connection header received
0618 22:31:30.787044 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.787048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.787103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.787156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.787742 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54946: BlockingRecv error: recv got EOF from 127.0.0.1:54946 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260618 22:31:30.807758 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.806501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.16.243.189:42307)
0618 22:31:30.806655 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.806661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.806928 (+ 267us) server_negotiation.cc:407] Connection header received
0618 22:31:30.807059 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.807062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.807112 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.807179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.807638 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54952: BlockingRecv error: recv got EOF from 127.0.0.1:54952 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260618 22:31:30.843196 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.841808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.16.243.189:42307)
0618 22:31:30.841931 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.841936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.842184 (+ 248us) server_negotiation.cc:407] Connection header received
0618 22:31:30.842318 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.842322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.842372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.842454 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.843063 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54962: BlockingRecv error: recv got EOF from 127.0.0.1:54962 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260618 22:31:30.910374 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:30.909269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.16.243.189:42307)
0618 22:31:30.909404 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:30.909407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:30.909609 (+ 202us) server_negotiation.cc:407] Connection header received
0618 22:31:30.909689 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:30.909692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:30.909744 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:30.909821 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:30.910253 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54964: BlockingRecv error: recv got EOF from 127.0.0.1:54964 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260618 22:31:31.041587 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:31.040522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.16.243.189:42307)
0618 22:31:31.040680 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.040684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.040741 (+ 57us) server_negotiation.cc:407] Connection header received
0618 22:31:31.040902 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.040905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.040952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.041019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:31.041460 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54978: BlockingRecv error: recv got EOF from 127.0.0.1:54978 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260618 22:31:31.300594 7684 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:31:31.299634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.16.243.189:42307)
0618 22:31:31.299781 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.299786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.299806 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:31:31.299926 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.299928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.299977 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.300049 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:31:31.300439 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54986: BlockingRecv error: recv got EOF from 127.0.0.1:54986 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:31:31.615734 7799 catalog_manager.cc:1259] Loaded cluster ID: 7c27456a8664437bb75ea3f6141254e6
I20260618 22:31:31.615819 7799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: loading cluster ID for follower catalog manager: success
I20260618 22:31:31.617606 7799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: acquiring CA information for follower catalog manager: success
I20260618 22:31:31.618845 7799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74cd8eef402f45cbafa71bef9303de50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:31:31.623085 7812 catalog_manager.cc:1259] Loaded cluster ID: 7c27456a8664437bb75ea3f6141254e6
I20260618 22:31:31.623162 7812 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: loading cluster ID for follower catalog manager: success
I20260618 22:31:31.624936 7812 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: acquiring CA information for follower catalog manager: success
I20260618 22:31:31.626144 7812 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 389219b5bfe94af3b56799c2eab1e67f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:31:31.818598 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:31.814595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.16.243.189:42307)
0618 22:31:31.814834 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.814838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.814853 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:31:31.815010 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.815014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.815143 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.815247 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:31.815539 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.816120 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.817077 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.817304 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.817380 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:31.817690 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:31.818167 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:31.818256 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:31.818337 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 22:31:31.818390 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260618 22:31:31.825244 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:31.821589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.16.243.188:36673)
0618 22:31:31.821820 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.821825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.821840 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:31:31.821983 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.821985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.822113 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.822213 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:31.822421 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.823012 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.823838 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.824077 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.824161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:31.824442 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:31.824864 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:31.824972 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:31.825038 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 22:31:31.825093 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260618 22:31:31.832619 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:31.828387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.16.243.188:36673)
0618 22:31:31.828501 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.828505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.828572 (+ 67us) server_negotiation.cc:407] Connection header received
0618 22:31:31.828746 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.828749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.828800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.828861 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:31.829146 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.829904 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.831347 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.831683 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.831783 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:31.832127 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:31.832214 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:31.832281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:31.832416 (+ 135us) server_negotiation.cc:299] Negotiation successful
0618 22:31:31.832476 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:31:31.832851 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:31.828167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.16.243.189:42307)
0618 22:31:31.828317 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.828321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.828674 (+ 353us) server_negotiation.cc:407] Connection header received
0618 22:31:31.828896 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.828900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.828954 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.829033 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:31.829287 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.830057 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.831848 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.832103 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.832197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:31.832405 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:31.832477 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:31.832537 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:31.832670 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 22:31:31.832726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:31:31.833328 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:31:31.828077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.16.243.190:41381)
0618 22:31:31.828316 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0618 22:31:31.828321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:31:31.828650 (+ 329us) server_negotiation.cc:407] Connection header received
0618 22:31:31.828752 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:31:31.828755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:31:31.828945 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:31:31.829046 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:31:31.829361 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.829975 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.831315 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:31:31.831684 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:31:31.831783 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:31:31.832405 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:31:31.832774 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:31:31.832848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:31:31.833127 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 22:31:31.833192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260618 22:31:31.840340 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7535
I20260618 22:31:31.845806 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7610
I20260618 22:31:31.850657 17358 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9_akNk/build/debug/bin/kudu with pid 7685
2026-06-18T22:31:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3005 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5966 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275123 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST